I'm hoping someone here can give me a little nudge in the right direction.
I have a Drupal site using quite a few contrib and custom modules. I am running APC with cacherouter and have my page load times down around 1000 ms, with < 100ms in the db (stats reported by devel/performance). Not bad, but I have a pretty fast server and want to get that a little lower. About 20% of the time the time is around 4000ms, with the same time in db. In addition, every once in a while I get a very long load time, like 45 seconds, again with the same time in the db. Memory usage for the page varies from 30MB to 60MB. That seems high to me, but there are quite a few modules so maybe that's to be expected.
Here's an example of a slow page load:
Page execution time was 12700.99 ms. Executed 211 queries in 59.42 milliseconds.
Memory used at: devel_init()=2.02 MB, devel_shutdown()=33.82 MB.
So I am trying to see where all of that 1000 ms+ is happening. I started by looking at the module load times using the method here: http://2bits.com/articles/high-php-execution-times-drupal-and-tuning-apc.... 23ms for all the modules, so OK here. Next I profiled with xdebug and wincachegrind, and the total times are in the 40ms range. Now I'm stumped; where is all of that extra time coming from?
When I look at the cpu and memory usage of the server and vm containers it's not taxing the server too badly; there is at least 50% memory and CPU headroom at all times. APC is set to use 128MB, has at least 50MB headroom.
For reference, here is some info about the setup:
Server:
Dell 1850, 3.65Ghz Dual HT, 4GB RAM, 2x80GB 15krpm SAS
Centos 5 64 bit running OpenVZ
Drupal Container:
Ubuntu 9.04 64 bit
PHP 5.2.6
Drupal 6.13
MySQL Container:
Centos 5 64 bit
MySQL 5.0.45
Am I missing something fundamental about php script execution times? I would expect that what drupal is reporting would match up with xdebug...
Thanks,
-Aaron
Comments
From my knowledge PHP opcode
From my knowledge PHP opcode caches don't work well with Xdebug profiling, i.e. the reported times get all wrong. Not sure if this is still valid, though.
-Stefan
Thanks, I will check that out
Thanks, I will check that out
Things to check
There are several things to check on any site that displays the same symptom (total page execution time is high but database query time is low).
Are you using views? Views does not report its queries in the devel's queries, so the database figure from devel can be misleading. You can either use views 6.x-3.x-dev or the patch in this issue (works with some 6.x-2.x versions) http://drupal.org/node/631522
Does your site do network connections from the backend (i.e. over PHP)? If so, this can eat a lot of time, and if the other site is down your site can hang there. There are quite a few modules that do that, mainly for APIs and widgets to other sites. See an example here http://2bits.com/articles/how-relying-connections-third-party-servers-ca...
Does your site send emails immediately when content is posted? This is a variant of #2 above, but if you have thousands of users that the email go to, then it will take time and eat up resources. Better defer that to cron. See here http://2bits.com/articles/how-reduce-page-load-times-sending-emails-cron...
Do you have all your site's file in NFS, or on a shared disk? When you have lots of modules, each module loaded for every page view, or at least an lstat() call is done on it. On a remote file system like NFS.
Do you have enough memory allocated for APC? Sometimes if you don't you get fragmentation that really really hurts performance. See more here http://2bits.com/articles/importance-tuning-apc-sites-high-number-drupal...
Try those and post back.
Drupal performance tuning, development, customization and consulting: 2bits.com, Inc..
Personal blog: Baheyeldin.com.
Thank you very much for Views queries tip.
Very helpful!
Much Read-Only?
Is much of the data read-only, that can be kept in an in-memory cache via Squid or Varnish? I followed some instructions for building a Project Mercury instance at http://groups.drupal.org/node/25425 on a shared server last week. I got the page load from a second or two to just leaping on the page. I didn't do any formal measurements, but had an impressed client.
Images in particular are an easy win here. To really squeeze out the performance read up on the Varnish docs about edge side includes. It's truly wonderful stuff.
Update, looking pretty good!
Thanks for all of the suggestions, here is my report after spending some time looking in to these, as well as a brain dump of everything performance-related that I have dealt with:
First, one thing that bit me and explains one of the issues with long page execution times: I had a couple of different apache config files that I was experimenting with and my dev version had the default concurrent connections of 10 set. Apparently I merged that one into my production config at some point. I caught it before the site launch but that was a big DUH moment! What led me to this was the Net tab on Firebug saying that some requests had very long queue times, like 5-7 seconds while I was loading the site. My server debugging was indicating that the page was rendered fairly quickly but there was still a huge lag. Interestingly, this change seemed to lower the memory usage per PHP process significantly. Each is running under 30MB now. Not sure if those are actually related. FYI I had to increase the total memory available to the container when I increased the number of allowed requests, so be careful with that setting. One note here: concurrent connections isn't concurrent users for most sites. For example, on my site some pages make up to 5 requests that must be processed by php, so I have set the concurrent connections setting in apache to about 3x the number of concurrent users and that seems to be working pretty well.
Views: thanks kbahey for the tip on views. I applied the patch to get their execution time into the devel total and indeed there was significant time spent in the db for the views. My site is mostly static content that varies only slightly by role, but none of the views do, so I turned on caching of the views db results and their rendered output and cut about 120ms off of the average page load.
Emails: emails are all on Cron daily digest, except for forgot password... great tip but I had already done that for other reasons.
APC: I forgot to write in the original post about the APC settings. After installing cacherouter I did bottom out the 48MB I had allocated bump up the size of the APC cache to 128 MB. It's running about 80MB now with good hit/miss ratio of over 98% cache hits. One strange thing about this: I read on 2bits and several other sites about apc stat and how you shouldn't turn that on unless you aren't updating any scripts on the site. I had that on on the original config on my dev box and never seemed to run into this, which perplexes me a little. I was definitley doing all kinds of tweaking of php files and seeing immediate changes on the server. Also thx Stefan Kudwien for the tip about opcode caches and xdebug. Indeed turning off APC while profiling showed different results, but I'm not sure how I would debug a complex issue since I would ideally like to see
File System: File system is local and very fast, so no issue here. One thing to note on the subject of disks though, and this has bitten me more times than I can count: It's notoriously difficult to get feedback on the status of your disk arrays while under normal operations, like without booting to bios. I have dealt with this with all flavors of Windows, Centos, and Ubuntu, on Dell, IBM, and hand-rolled servers. The Raid vendors only "fully support" very specific hardware and os configurations, so if you have a choice and you're setting up your own hardware, try to set yourself up to be able to get alerts within the OS. I can't count how many times I have had a site go down because of an array failure that I could have prevented if there was some kind of reporting of degraded drives to the OS where I could monitor it. You might think that the R in RAID is providing you with some security but it's really amazing how often RAID 0 volumes fail. SOrt of a digression from the topic at hand but something to think about.
Network connections to other sites: not currently using any but good point. One thing that I ran into on another site that's sort of related: Google Analytics. We ended up caching the GA js files ourselves because of some severely degraded response times from google. I noticed that in the GA Drupal module that there is a similar caching setting.
OK so I'm pretty happy with where I'm at now:
~200ms render times for anonymous pages
<1s render times for authenticated pages
Nothing unexpected in terms of really slow pages, qeueing, or server resource allocation
Good visibility into site performance and ability to debug/analyze data (apc.php, xdebug/cachegrind, devel modules, performance module, views patch to get view execution times into devel count, Firebug Net tab, YSlow)
Next steps:
Check out Pressflow/Varnish/Squid: http://fourkitchens.com/pressflow-makes-drupal-scale
Internally optimize my custom modules and remove / combine modules if possible to lower total number of modules
Thanks to everyone who replied, I really appreciate your time on this!
-Aaron
CDN
Setting the number of concurrent processes to 3x the number of concurrent users sounds like a bit much. Not yet running Drupal on my main site but for phpBB I have two webservers with max_clients set to 60 with 2000 concurrent users (5 minutes). Normally only 5 to 10 processes are active on those servers.
There are some important differences between our setups though.
Firstly all my pages are generated in under 200ms, most under 100ms. This makes a huge difference in serverload ofcourse.
The other important difference is that I have a separate server for static content (javascript, images, css, etc). This makes a huge difference, it's pretty inefficient to server out static content with pretty heavy Apache processes that include PHP and a bunch of other modules.
In my case I'm using squid on a static.example.com subdomain, squid just reverse proxies all static content. I.e. www.example.com/images/logo.png is also available as static.example.com/images/logo.png There are several CDN modules/solutions available for Drupal.
You don't even need a separate box for this, just using Nginx, varnish, squid or even a stripped down Apache can make the difference. You don't even need changes to Drupal if you put something like varnish in front of Apache in your current setup.
Oh, and it helps to disable Keepalive on Apache or set it to something like 1 or 2 seconds.
Since you're running APC, and
Since you're running APC, and all PHP is therefore in the opcode cache, combining your custom modules will probably have no noticeable effect.
--
Dave Hansen-Lange
Director of Technical Strategy, Advomatic.com
Pronouns: he/him/his