Posted by peterconnolly on February 1, 2010 at 2:34am
I've got a server performing badly at the moment - giving 500 Internal Server error every couple of page refreshes. There are just over 1700 users online at present.
I've checked top, and it's showing me six occurrences of the httpd command using over 10% of the CPU each - with one or two being over 20%, and occasionally over 30%. I've attached a snapshot of the top command in case there's useful information on there I've missed off.
Help! Where should I look for more information on this to find out why this CPU usage is high, and how can I bring it down so that the server is more responsive?
Regards,
Pete
| Attachment | Size |
|---|---|
| top.jpg | 117.39 KB |
Comments
Odd. What OS is this? Linux
Odd.
What OS is this? Linux top shows the wait CPU time (time the CPU is waiting for something else to happen before it can do more work - usually I/O). Since your CPU is being reported as idle, but these HTTP requests are together using 146% it might be an I/O issue. How many cores do you have? Does your site make external connections to something? Are your disks going crazy? Or is Drupal actually using that much CPU? If so is there anything useful in the Drupal or HTTP logs?
--
Dave Hansen-Lange
Director of Technical Strategy, Advomatic.com
Pronouns: he/him/his
Linux
Thanks for the quick reply Dave.
Operating system is FreeBSD 7.1-STABLE running on an Intel Xeon quad core processor. There are no external connections - MySQL runs on this server.
How can I tell if the disks are going crazy, or if Drupal is actually using all that CPU? (I have SSH access, but the server itself is colocated miles away)
Memcached is running on this server, with the following stats;
Memcache Server version: 1.4.4
Process id of this server process 46189
Number of seconds this server has been running 113543
Accumulated user time for this process 11.645539 seconds
Accumulated system time for this process 45.063953 seconds
Total number of items stored by this server ever since it started 1673
Number of open connections 7
Total number of connections opened since the server started running 296678
Number of connection structures allocated by the server 20
Cumulative number of retrieval requests 798166
Cumulative number of storage requests 1673
Number of keys that have been requested and found present 483190 (60.538%)
Number of items that have been requested and not found 314976(39.462%)
Total number of bytes read by this server from network 53.506300926208 Mega Bytes
Total number of bytes sent by this server to network 3039.2949571609 Mega Bytes
Number of bytes this server is allowed to use for storage. 64 Mega Bytes
Number of valid items removed from cache to free memory for new items. 0
The Drupal log is showing the following error (and a lot of them...)
Type php
Date Sunday, January 31, 2010 - 19:09
User admin
Location http://mmmmmmmmmm.com/nodecarousel/fetch?start=5&name=content_rotator&aj...
Referrer http://mmmmmmmmmm.com/users/admin
Message include_once() [function.include]: Failed opening 'modules/user/user.admin.inc' for inclusion (include_path='.:/usr/local/share/pear') in /usr/local/www/data/includes/theme.inc on line 283.
Severity error
Hostname 24.10.199.94
The error.log is showing
[Sun Jan 31 19:08:49 2010] [error] [client 24.10.199.94] PHP Fatal error: require_once() [function.require]: Failed opening required './sites/all/modules/cck/theme/theme.inc' (include_path='.:/usr/local/share/pear') in /usr/local/www/data/sites/all/modules/cck/content.module on line 177, referer: http://mmmmmmmmmm.com/users/admin
followed by many occurences of
[Sun Jan 31 19:20:47 2010] [error] [client 85.114.136.243] File does not exist: /usr/local/www/data/sites/default/files/images/magazine/article-promotion-current-<br
I don't know why the ' <br' is at the end of this message; the image would be article-promotion-current-issue.jpg.
Thanks in advance
Pete.
more information needed
Hi Pete.
You just use top for CPU/memory review. There are fstat, gstat, iostat, nfsstat, pstat
To see if you disks load are ok you can start with iostat.
My suggestion is to install and configure some sort of "monitoring" tool, like munin
do you use boost or cacheroter?
which Drupal version you use?
do you use php accelerators?(APC, eacceleratol, xcache)
do you install php/apache/mysql from ports or compile them by hand? which options you use?
@alexandr iostat gives the
@alexandr
iostat gives the following results;
I tried the other commands, but they don't appear to be installed on this server.
I'd love to get some useful monitoring on this server - will investigate munin, but I won't have permission to do that tonight.
The server has nginx/0.7.64 installed, with memcache.
Drupal is 6.15
No PHP accelerators.
php/apache/mysql installs are done by the server team - I've just been given the problem to solve! I suspect they are the standard FreeBSD installs.
Guess is I/O issues
No PHP accelerators.Because the php code is not in memory this can slow you down a lot. Highly recommend getting something like APC running.
APC is now installed
I've had APC installed, and saw a small increase if memcached was switched off.
If I switched memcache on, speed dropped dramatically - not what I'd hoped to see with memcached.
Whilst I was SSH'd in, I ran the command
ab -n1500 -c5 http://xx.xx.xx.xx/ (the ip address of the server)
With APC installed in PHP (do I need to do anything else?) and Memcached off, I saw an average of 235 requests per second. This was inline with the scores I saw before APC was installed (we had xcache on, I've found out).
Once I turned on memcached along with APC, the average dropped to 14 requests per second.
I was rather hoping that memcache would improve things, but that doesn't seem to be the case.
Regards,
Pete.
Check the version of your
Check the version of your PECL memcache package (You can find it in phpinfo). Sounds like you have an older version. I went through this exact same thing on a FreeBSD server. We upgraded the memcached servers to 1.4.4 but our PECL memcache package was still at 2.2.1. We ended up upgrading it to 2.2.5 and everything was happy again.
HollyIT - Grab the Netbeans Drupal Development Tool at GitHub.
APC requires RAM to be
APC requires RAM to be efficient, and so does memcache. If running APC & memcached caused swapping on your server, it might explain such a bad result. To monitor swapping in real time run "vmstat 1 1" on you server while you are testing.
APC is an opcode cache, it can't possibly be the blame, and memcache is a very fast memory cache - I can't see how it can be blamed either.
I suggest to look for the problem around, it can't be memcache/APC by themselves.
APC configuration
Per earlier post, make sure APC has enough memory to be effective, check the hit rate - you should be able to achieve close to 100%
Dev and Support: prometsource.com
Thanks!
@intoxination: I did a phpinfo, and memcache is 1.4.4, the following is the output for memcache in phpinfo;
memcache
memcache support enabled
Version 3.0.4
Revision $Revision: 1.83.2.36 $
I see no mention of PECL memcache... is that something else I need to install?
@yhager: I don't see any evidence of swapping on the server - Swap is 4096Mb, with only 288Kb used.
Thanks!
Pete.
Your pecl package (what
Your pecl package (what allows PHP to communicate with the memcache server) is 3.0.4. The entire 3.0 branch is still in beta. I would downgrade it to 2.2.5 and see if that helps. That's what we ran on FreeBSD and never had a problem since.
HollyIT - Grab the Netbeans Drupal Development Tool at GitHub.
Can't downgrade, I've been told
@intoxination: I spoke to the server chaps, and they said that the version of pecl-memcache that comes with the operating system is 3.0.4, and so they can't downgrade it. :-(
http://www.freebsd.org/cgi/ports.cgi?query=pecl-memcache&stype=name
Why Beta software would be released, I don't know.. but then again, there's a lot I don't know... :-)
Pete.
An update
I feel that memcache on the existing server was having problems - probably related to intoxination's comment about the pecl-memcache still being in beta.
I spoke to my regular server people in the UK, and they gave me an equivalent spec dedicated server to play with for a week. Without too much drama I was able to load the existing site and database on there, get memcache/apc and boost installed, and the test site scored over 3,300 requests per second on ab, and 99.736% cache hits. A significant increase from the 14 requests per second and 60.5% cache hits the live server was giving.
We can now talk to the client about spending more time finding out what was wrong with the live server, or just move everything to a new server. Thanks to everyone here for their help in nailing the issues down.
Regards,
Pete