To follow on from my last post, having data showing the load on your server throughout the day is great, but what do you do with it and how do you interpret the information?
Looking through the data I could see clear short periods where the load average would sky rocket, into double figures and into the 20's. From every article and post I'd seen it said that if the load divided by the number of cores/threads was more than one you have a problem. No ifs, no buts, you have a problem. I was seeing 20+ on a dual cpu server... oh dear!
The problem was I couldn't see an obvious cause. CPU usage wasn't high, the idle time was good, memory was fine with the swap file not being used, the disk queue wasn't long, and the process list didn't show anything to indicate an issue.
Fortunately I came across a few fantastic explanations of Load Averages that explained where I (and it seems many others) had been going wrong. I've linked to them all below, and I recommend reading them for more info, but the upshot is that it's not as cut and dry as people make out.
To quote from Jon Emmons blog, the load average "is the average sum of the number of processes waiting in the run-queue plus the number currently executing over 1, 5, and 15 minute time periods.".
The load average is far more complex than many people make out, and while it can be a good initial indicator of a problem it must be examined in conjunction with other factors. It doesn't allow for the fact that a process could be waiting for not just the CPU, but also disk or network IO, and doesn't allow for the priority of the running/waiting processes in the queue.
If you have a long running low priority process running for instance, that will always make way for more urgent time critical requests. In the mean time that process will sit in the queue, and will cause the load average to increase. Add some more of these low priority processes, for instance a backup job, and the load average will increase, which indicates a problem. Higher priority processes like email, websites etc will be handled immediately however, causing no delay for users, and as such the reported high load isn't really an issue.
So the key is that it's fine to track the load average to indicate a possible problem, but don't rely on it for proof that you have one. Always remember to check the other figures provided to see IF you have a problem, not necessarily WHAT the problem is.
http://www.lifeaftercoffee.com/2006/03/13/unix-load-averages-explained/
http://blog.mellowhost.com/confusing-server-load-average-explained.html
http://www.teamquest.com/pdfs/whitepaper/ldavg1.pdf
http://www.teamquest.com/pdfs/whitepaper/ldavg2.pdf
Showing posts with label Server Load Average. Show all posts
Showing posts with label Server Load Average. Show all posts
Sunday, 12 August 2012
Tuesday, 7 August 2012
Tracking Linux server load and processes over time
Tracking down the cause for high load on a server can be a challenge, especially when issues are happening when you can't keep an eye on it, but it's a challenge I've been facing recently. While there were no issues with the server's response, I discovered at certain times (often in the evenings) the load average would become high enough to cause Exim to briefly pause processing the mail queue.
After hunting round for a easy way to track the server's state I came across a simple script from Craig Edmonds that did the job. It very simply generates an email containing a variety of status information, including most importantly the process list data from Top, and sends that to you in an email. By scheduling the script to run every minute you get a snapshot of the servers state at regular intervals. Since the subject line includes the load average you can easily look through the messages, spot those times with a high load, and see what the server is doing.
In my case I adjusted the script slightly to include $todaydate in the subject line, since the above issue with Exim meant I couldn't always rely on the message being received in the correct order.
There was one problem I found with this solution. The script runs a single iteration of Top and inserts the output of that into the email, however as you can see from the Man page for Top :
The top command calculates Cpu(s) by looking at the change in CPU time
values between samples. When you first run it, it has no previous sam-
ple to compare to, so these initial values are the percentages since
boot. It means you need at least two loops or you have to ignore sum-
mary output from the first loop. This is problem for example for batch
mode. There is a possible workaround if you define the CPULOOP=1 envi-
ronment variable. The top command will be run one extra hidden loop for
CPU data before standard output.
each email I received had identical CPU data. While I could tell the server's load was high, I couldn't see what the state of the processor was at that time. I didn't fancy messing around with environment variables, so instead opted for a solution found here, and adjusted the line calling Top as follows :
$process_list = shell_exec('top -b -n2 | awk "/^top/{i++}i==2"');
So I found a simple and easy way to track what's happening on the server, though of course with an email a minute it's not something I'll be running long term.
I wish I could say that was the end of it, but unfortunately this turned out to be the beginning of my struggle and confusion, brought on in no small part to the number of confused explanations of Load Average operations, but I'll discuss that in my next post.
After hunting round for a easy way to track the server's state I came across a simple script from Craig Edmonds that did the job. It very simply generates an email containing a variety of status information, including most importantly the process list data from Top, and sends that to you in an email. By scheduling the script to run every minute you get a snapshot of the servers state at regular intervals. Since the subject line includes the load average you can easily look through the messages, spot those times with a high load, and see what the server is doing.
In my case I adjusted the script slightly to include $todaydate in the subject line, since the above issue with Exim meant I couldn't always rely on the message being received in the correct order.
There was one problem I found with this solution. The script runs a single iteration of Top and inserts the output of that into the email, however as you can see from the Man page for Top :
The top command calculates Cpu(s) by looking at the change in CPU time
values between samples. When you first run it, it has no previous sam-
ple to compare to, so these initial values are the percentages since
boot. It means you need at least two loops or you have to ignore sum-
mary output from the first loop. This is problem for example for batch
mode. There is a possible workaround if you define the CPULOOP=1 envi-
ronment variable. The top command will be run one extra hidden loop for
CPU data before standard output.
each email I received had identical CPU data. While I could tell the server's load was high, I couldn't see what the state of the processor was at that time. I didn't fancy messing around with environment variables, so instead opted for a solution found here, and adjusted the line calling Top as follows :
$process_list = shell_exec('top -b -n2 | awk "/^top/{i++}i==2"');
So I found a simple and easy way to track what's happening on the server, though of course with an email a minute it's not something I'll be running long term.
I wish I could say that was the end of it, but unfortunately this turned out to be the beginning of my struggle and confusion, brought on in no small part to the number of confused explanations of Load Average operations, but I'll discuss that in my next post.
References :
http://www.unix.com/gentoo/77494-top-batch-mode-cpu-info-wrong.html
Subscribe to:
Posts (Atom)