Hi folks. I'm looking for some feedback and advise on a odd performance issue.
We have a Drupal 6 multisite that we've developed that currently houses 7 sites. The sites themselves are, as Drupal sites go, not particularly exciting or complex. It even has fewer modules than most sites we build. (All of the sites use the same modules and configuration.)
On our development server, performance is fine.
On the client's test server, performance is fine.
On the client's live server, performance is completely unacceptable and horrific. Some pages easily take over a minute to load. The modules page times out, despite the server being configured to allow 128 MB of RAM per PHP process. The performance is a nightmare both on the front end (using a custom theme) and the backend (using Garland), so it's not a theme-related issue. It is even terrible loading update.php, so I'm skeptical that it's even related to module code at all.
The live server is running an opcode cache, while the dev and test servers are not. Now I would blame higher traffic on the live site, except that the same physical box is hosting a number of other Drupal installs, including a D5 multi-site that we built, which are running just fine. I'm also not sure that the sites are even published, so the actual traffic to them is minimal.
The client is trying to blame the lack of a favicon file in the docroot, or the fact that the favicon's permissions keep changing (although I am unclear as to why that would happen). I know a missing favicon can cause buggy versions of IE to generate lots of stray 404 requests, but that (a) was fixed in an earlier D6 release and (b) is definitely not going to cause 60-second page load times.
The only remotely suspicious notes in watchdog are warnings that
The specialized Panels menu router rebuild failed. Some paths may not work correctly.
When the cache is flushed. However, nothing seems to be not working as a result of it.
At this point I'm at a loss. What would be causing this sort of slowdown? What sorts of things should we be looking for?
Comments
I am guessing that you have
I am guessing that you have too many menu rebuilds. Maybe the menu cache isn't being built correctly at all or some stupid code tries to rebuild it since it thinks it is broken.
Try putting
'menu_rebuild_needed' => FALSE,into your $onf array in settings.php.
Tried, no change
Thank, killes. That doesn't appear to be having any effect, though. Even with a menu rebuild on every page, though, I can't see that causing a 20 second page load time. Menu rebuild isn't that expensive.
I've found menu rebuilds to
I've found menu rebuilds to be very expensive, but I guess it depends on which kind of menu you are talking.
devel?
Have you tried devel with query log on the live site vs. the test site(s) and see what is different and where the time is spent?
You can also enable the "performance" module in summary mode on the live site and see what it tells you.
Drupal performance tuning, development, customization and consulting: 2bits.com, Inc..
Personal blog: Baheyeldin.com.
Drupal performance tuning, development, customization and consulting: 2bits.com, Inc..
Personal blog: Baheyeldin.com.
Devel results
Well I tried reloading the front page a number of times, and the results I got were all over the place. The fastest was this:
But I also got this, and a few worse:
Most of the slow queries are cache_get()s, except for one bad looking view. But that view wouldn't account for an 11 second page load.
The final sess_write() call is also sometimes taking over 200 ms, despite the entire query being quite small. There's a few other queries that occasionally spike, which IME is normal for MySQL. Bah.
It seems like the performance is varying over time. At some points it's around 5 seconds per page load, others it's up near 20 routinely. That's despite us making no changes during that time. That's what leads me to question it being a server issue rather than a Drupal issue.
Rules out the database
Well, the database shows some variability (337 vs. 1450 ms), but the bulk of the time is spent in PHP (compare at 4215 and 11,580).
Are you sure the site is an exact copy of the test site? Does it do any "over the network" stuff, like contact another host to get something via fsockopen() or something like that? netstat should tell you that.
In such cases, I either use strace from the command line to see what is going on, or if xdebug is available, make it dump a file that I can use kcachegrind on to see where the time is being spent.
Drupal performance tuning, development, customization and consulting: 2bits.com, Inc..
Personal blog: Baheyeldin.com.
Drupal performance tuning, development, customization and consulting: 2bits.com, Inc..
Personal blog: Baheyeldin.com.
Something else on the server eating resources?
RAM? CPU? Pagination? Slow disk? ...well, it's just an idea. You haven't metioned anything about the hardware and other services running on the servers. Maybe the difference is there.
Caching
If you have a lot of authenticated users accessing the site, then you are encountering the caching/scalability issue.
However, if it is more unauthenticated users, then you might want to consider looking at installing the APC and Memcached PECL libraries along with CacheRouter. Another option would be to install the Boost module as well.
CacheRouter: http://drupal.org/project/cacherouter
Boost: http://drupal.org/project/boost
Could you post/attach a copy of your PHP's php.ini, MySQL's my.cnf, Apache's httpd.conf and an output of 'phpinfo()' as well of the server environment? Perhaps it might be a couple of variables that need to be tweaked for scalability needs.
Authenticated Users
What counts as a lot? Let's say you have a total of 7 on average? That is not a lot correct?
Can you paste output from
Can you paste output from top?
Sometimes there's issues with APC being misconfigured - allocating too much memory (64MB is about right, 256M can start to slow things down) - or two little might cause fragmentation as well. I can't see that causing 20 second delays though. stick apc.php in the web root and it's usually pretty obvious if something's wrong.
Fragmentation, but unlikely
I thought about APC fragmentation and how it can negatively affect a Drupal site performance. But he has 128M allocated per process, and I have yet to see a site that needs more than that.
Larry, you can also experiment with APC's include_once_override. It may be of some benefit.
Drupal performance tuning, development, customization and consulting: 2bits.com, Inc..
Personal blog: Baheyeldin.com.
Drupal performance tuning, development, customization and consulting: 2bits.com, Inc..
Personal blog: Baheyeldin.com.
are the tables innodb or
are the tables innodb or myisam? are you using the book module? have you looked at the patches over at tag1consulting.com. feel free to contact me if you need help evaluating the site. can you run sometime like mytop and figure out if some tables are stuck because they are locking? also i would check to see how many users are in the sessions table. maybe time to convert sessions and users to innodb at minimum.
Slantview Media http://www.slantviewmedia.com/ | Blog http://www.slantview.com/
Not the database
I don't think it is the database.
If it was the database, then the time for queries would be the largest percent of the total page execution.
But it seems here that the PHP time is the high one, with the database making up only 8 to 12% of the total time, which is quite odd. Also the variance is quite high from one page load to the other, which says something else is going on.
What makes "odder" is that the site may not even be getting live traffic (not advertised publicly yet)?
Drupal performance tuning, development, customization and consulting: 2bits.com, Inc..
Personal blog: Baheyeldin.com.
Drupal performance tuning, development, customization and consulting: 2bits.com, Inc..
Personal blog: Baheyeldin.com.
Hi, on a side note, why do
Hi, on a side note, why do you mention the book module? Is it particularly slow?
I mention the book module
I mention the book module cause it does a menu rebuild on every page edit which typically can cause a lot of menu rebuilds which has a race condition in Drupal 6.
Slantview Media http://www.slantviewmedia.com/ | Blog http://www.slantview.com/
anything in the php error
anything in the php error log by any chance that sticks out?
Definitely PHP
An update: I put together a test script that just runs some PHP burn loops. No database, no output, no Drupal, nothing. Just lots of PHP execution. On our test server it ran in 1.1 seconds. On the live server it takes upwards of 25 seconds. So it's definitely an issue with PHP, and not with Drupal or any Drupal modules.
I don't have admin access on the server so I can't check logs or try tweaking settings. If anyone has general PHP performance suggestions I can pass those on to the client. What's most weird is that there are other sites running on other vhosts on the same box, and those do not have this problem.
Hmmm
Strange that other vhosts are running well ...
Are the other sites static (plain HTML)?
If the loop is pure PHP (i.e.
$i++;) and nothing else, then check the CPU load on the machine.Run top and/or vmstat on the server and see if it is CPU starved.
What is the configuration of live server? Is it a Sempron for example with lots of stuff running? Is there something hogging the CPU?
Can you run strace on your test PHP script and capture the output?
strace -o /tmp/debug.txt php yourscript.phpWeird indeed ...
Drupal performance tuning, development, customization and consulting: 2bits.com, Inc..
Personal blog: Baheyeldin.com.
Drupal performance tuning, development, customization and consulting: 2bits.com, Inc..
Personal blog: Baheyeldin.com.
Comparing output of phpinfo() ?
If it's just CPU load in pure PHP processing, maybe sites in different vhosts use different PHP settings? Is it PHP module or something else? Do all vhost share the same configuration? ...it's really hard to tell.
multiple Opcode precomilers
Check to see if you have xcache.ini, apc.ini, or eaccelerator.ini in the/etc/php5/conf.d folder. I had a problem with xcache being declared in conf.d and eaccelerator being declared in php.ini. This caused all kinds of problems inclding seg faults.
Eric
I've just run into a similar
I've just run into a similar issue. It appeared overnight with no apparent changes as far as me adding new modules, etc. I'm getting painfully slow page loads now. Here is an example.
Executed 820 queries in 292.53 milliseconds. Queries taking longer than 5 ms and queries executed more than once, are highlighted. Page execution time was 59751.37 ms.
All of the other sites in the shared IP are operating at normal speeds. Not exactly sure what to do here, but the natives are getting restless.
PHP Profiler
Well the queries are fast, so a PHP profiler should help you figure out where the bottleneck is. That is still a lot of queries, though.
Just because you didn't change a module or enable / disable anything doesn't mean a module isn't at fault. If it worked before, it may have only been fast with a smaller data set, and you've steadily been accumulating data until you noticed a slowdown. You may also be hitting memory limitations.
Ken Winters
www.coalmarch.com
Ken Winters
New views? Quicktabs?
Did you create new views?
Do you have Quicktabs?
Sometimes views "masks" the long running queries.
Sometimes it is the custom rendering functions in the theme that are slow, look for phptemplate_preprocess_views_view() with node_load() and user_load().
We solved a case like this for a client the other week. You can see the details at Case study: views quicktabs and template.php.
The other thing to look for is operations over the network, for example service links module, or having an add served from PHP via a socket to another site.
Here is a case study on this: how relying on connections to third party servers can be detrimental to performance.
Drupal performance tuning, development, customization and consulting: 2bits.com, Inc..
Personal blog: Baheyeldin.com.
Drupal performance tuning, development, customization and consulting: 2bits.com, Inc..
Personal blog: Baheyeldin.com.