Drupal using 100% CPU

Posted by tjb74 on March 4, 2010 at 3:34pm

Hitting any link or doing any operation on the site and Top shows the server using 95 to 100% of the CPU to serve the www-data.

Firstly I am not a developer, I am just a photographer with an idea for a site, which because I couldn't find any local cms developers who would build it on Drupal, I started learning on my own. For the moment this is not about high performance, its about ANY performance!

I have built the site based on looking at functionality of similar sites that are successful and have very high traffic, but it uses 100+ modules to achieve similar functionality, and every page uses several views and blocks. (I also hope to have a popular and very high traffic site...)

I have APC php caching, I am using Boost, I have read up about and tried MySQl tuning.

An average page and Devel gives me: Memory used at: devel_init()=4.25 MB, devel_shutdown()=86.21 MB, but I have seen shutdown be regularly 123MB and goto 130+ MB

Another example to get the admin page Devel returns:
Executed 667 queries in 558.29 milliseconds. Page execution time was 19264.33 ms. Memory used at: devel_init()=4.25 MB, devel_shutdown()=119.72 MB.

The server is a dedicated server with 2GB of RAM, but I have another dedicated server running the same site with the same modules. It has 4GB RAM, but same thing Top shows nearly 100% CPU every time and the site seems slow.

With only a couple of users the server seems to grind to halt, and I am often locked out now it is 'live' for testing.

I am monitoring the server with Munin which gives great graphs, but I am not sure what results are good or bad? What should the Devel shutdown size be ideally? And how do you achieve it when you want a lot of functionality? What should reasonable CPU usage be? How many concurrent users would we expect for a site slow down or to be locked out?

I have been through my module list stripping them down, but as I am not a programmer I need to use them to achieve the functionality I want with my site using Drupal.

Anyone have any ideas why my Drupal installation could cause such high CPU usage and be so slow. Could it be a problem with a single module?

Thanks.

Comments

which processes took 100%? -

Posted by Alexander Kosarev on March 4, 2010 at 6:47pm

which processes took 100%? - it can be mysql, it can be web server - suggestions for each situation will be different.
- mysql can be slow because of bad tables structure and slow queries(munin mysql plugin can show which one is true)
- apache can be slow because of php 'calculation' caused by bad module logic

do you enable apache and mysql munin plugins? can you provide some stats for memory, CPU and I/O?

100+ modules - well, not all modules have good performance and even one of this modules can put your server down.

The division of processes

Posted by tjb74 on March 4, 2010 at 8:06pm

The division of processes jumps around a bit, but the high usage is the web server (www-data apache) taking the biggest chunk and up to 100%. Mysql doesn't seem to get over 30%, and where it does the web server will be at 70%.

I have mysql plugin for munin, but nothing is showing for slow queries, however I only installed it a day ago to try to nail down the problems.

What would the cpu usage be for the web server and mysql 'normally' or are there targets?
Is devel out memory usage high?

That is good to know a bad module might be the cause. I can start a process of elimination if that is the case.

Some things to check

Posted by ahb on March 5, 2010 at 2:05am

Sounds like you have a plenty powerful box that should perform even with lots of modules. With APC number of modules matters less in terms of php performance but if there is a bad one in there it can really hurt you so you should still probably look for that by process of elimination.

20 second page load time and 100% CPU seems to indicate some misconfiguration somewhere, or a very very badly performing module. In terms of what you should expect/shoot for, since you have a dedicated server don't worry yet about memory usage per thread. First try to get the pages to load in a reasonable amount of time. While your first inclination may be to install some caching modules that is not going to get you the performance you need, unless you have a content-only site with no user- or role-specific data, and even then, something is amiss so papering over it with caching is only going to hurt you later. So boost is of little use until your traffic is high.

Try these things and if you don't find that it helps you might look into hiring someone to remotely check out your config. I can't really recommend anyone but I know that there are companies out there who can help. Sometimes a well-placed $100 can save you days of hacking if you're inexperienced!

Make sure that APC cache is not set too small: put apc.php in your site root and browse to it. You should have some headroom in terms of unused apc memory and cache hits very close to 100%. If it's all orange you need to increase the size of apc cache, especially since it looks like some of your threads are using a lot of memory.
Apache ships with ridiculous settings for max concurrent users. Set MaxClients to something higher than default to avoid the lockout issue you describe. This one really perplexed me until I saw that apache ships with 10 as the default.
Devel does not show you the mysql time taken by views. There is a patch for views somewhere or maybe it's in the latest version. If you are really at 558 ms for db then your issue is most likely in a module or in php or apache configuration.
Try caching views output. There are caching settings on each view, so if you find that your views are slow you can cache. I have mine set to 5 minutes for a good balance between performance and update when new content arrives, etc.
Later on once you get the glaring problem under control you can optimize even further by using cacherouter to store drupal cache in apc instead of db. That can get you a little more performance but won't matter until your php time is closer to or less than your mysql time.

Good luck!

Apache bug

Posted by sreyas on July 16, 2010 at 5:40pm

May be this is related to the apache bug they have reported here https://bugs.launchpad.net/ubuntu/+source/apache2/+bug/230878

Check your /var/log/messages and see if those errors they list their comes in your logs also.

Regards
Sreyas

Implemented above given suggestions but still unable to fix

Posted by RavindraSingh on February 25, 2013 at 9:25am

I have implemented all above given suggestions for my drupal website but still facing 100% CPU usage while saving any content.

I thought this is a issue of apache which is using above then 90% of CPU usage.

Is there any other way to come out from this type of problem.

Diagnose, don't guess ...

Posted by kbahey on February 25, 2013 at 2:14pm

It is best if you go about this in a methodical manner.

Try to diagnose based on facts, not on guesses and assumptions.

Use a profiler such as xhprof or xdebug, and find out where the time is spent, rather than maybe this, maybe that suggestions ...

Drupal performance tuning, development, customization and consulting: 2bits.com, Inc..
Personal blog: Baheyeldin.com.

Agreed, guessing is not the

Posted by martijn houtman on February 25, 2013 at 2:28pm

Agreed, guessing is not the best way to go, while a profiler is.

I suspect either too little memory for APC (use the apc.php to check memory usage and fragmentation), or some (really) faulty module. In case of a faulty module, you'd be best off using some kind of profiler. I've always used xhprof and was able to determine what function was called often or took a long time rather easily.

Another (foolish) way would be to try and disable a few modules each time, and see if the speed improves. And I always use memcache (almost by default), but that only works if your caching is setup correctly (i.e. not with faulty modules).

Are you running mod_php? If

Posted by wipeout_dude on February 25, 2013 at 9:51am

Are you running mod_php?

If you are you really need to change this for another way of running php (There are at least 4 ways to run php and each have their pros and cons).. The reason I suggest this is because I imagine you have a very graphical site.. If running mod_php then apache will launch a full php stack for every request for every asset on the page.. This means high memory usage and load on the server..

I am not saying this is the cause of your problem but will help reduce the memory requirements on the server..

which are the 4 ways to execute the php

Posted by RavindraSingh on February 25, 2013 at 10:11am

Can you tell me which are the 4 ways to run the php without using mod_php in apache server.

I just wana to go through with your suggestion if it take me out from this issue.

Actually there are 5 ways..

Posted by wipeout_dude on February 25, 2013 at 11:39am

Actually there are 5 ways.. :)

mod_php - Apache module, full support for APC, large memory overhead for non-php assets delivered by the server..

cgi - Apache executes the php via a cgi process, once complete the process is destroyed.. Non-php assets and delivered from apache without any interaction with php.. Can't be used with APC because process (and cache) are destroyed after execution..

fcgid - Similar to cgi but keeps php processes (configurable) running.. In my tests still can't be used with APC effectively because each process is independent and can't share the cache..

fastcgi - Again similar to fcgid just slightly different in its management or processes.. Better for use with APC but still not great..

PHP-FPM - The newest and best way to run php in my opinion.. Addresses all the issues of the other ways and PHP becomes its own process manager separate from the web server.. Apache 2.4 and Nginx can directly communicate with PHP-FPM, on Apache 2.2 you will need to use something like fastcgi to interface with it..

Also look at using PHP 5.4 if you aren't already.. Its faster and uses less APC memory for the same cache..

One warning on PHP 5.4. Check

Posted by Jamie Holly on February 25, 2013 at 2:38pm

One warning on PHP 5.4. Check the issue queues of contrib modules to make sure there aren't any problems!

What I would do is head over to New Relic and sign up for the free account and put their monitoring agent on the server. It gives you a great view of which modules are taking up the longest run times and most function calls. That should help isolate the problem.

I manage a couple large Drupal installations for clients running mod_php and never have any high CPU usage problems.

If you're running APC, make sure that is updated. There have been versions out there that cause high CPU problems. I haven't seen any in a few years, but there's a chance you might have one of these problem versions.

Another common problem that causes this is cache stampedes (contention in CS terms). If you have a cached item that takes a lot of processing to build and 2 or more users hit the page requesting that item at the same time, then they all require a rebuilding of that item. Drupal 7 does have a locking mechanism to help prevent this. Drupal 6 also has a locking mechanism, but it isn't used for cache. Pressflow does have a locking mechanism in it for 6 and I believe cache router does as well. This prevents the same cache rebuild being run multiple times in parallel.

HollyIT - Grab the Netbeans Drupal Development Tool at GitHub.

I tried all the given suggestion.

Posted by RavindraSingh on March 4, 2013 at 6:22am

I tried all the given suggestion but i am utilizing cck in my drupal website. Which i can't disable. In my current drupal installation 3 user are using 90 - 95% CPU usage and if 4th user comes the it goes to 100% and that time mysql throw the 'PDO exception' or 'Savepoint ' like exception.

Did you try using a profiler

Posted by wipeout_dude on March 4, 2013 at 8:07am

Did you try using a profiler or New Relic to work out exactly where the problem is?

I have a similar issue, when

Posted by vako on January 2, 2014 at 5:36pm

I have a similar issue, when 5-6 people access the site the CPU usage goes up and my ISP is putting a block on the domain!
I have disabled all the unnecessary modules and still the issue is pending.
I will install New Relic but don't think it will help...:(