Posted by lighttree on August 2, 2012 at 2:21pm
Our site is experiencing some very poor page load times. According to Google Analytics, our average page load time is 10.22 seconds over the last month. Some pages like http://www2.buildinggreen.com/blogs/end-line-passive-solar are averaging over 22 seconds. I've also run a test on webpagetest.org which indicated that our average TTFB is 3-5 seconds.
We are using APC and Varnish and have caching enabled in drupal. What is the best way go about debugging an issue like this? Are there any recommendations for adjustments that could be made to apache, php or mysql that might improve this?
Any assistance with this would be greatly appreciated.
Comments
Install and enable
Install and enable http://drupal.org/project/devel
In the config enable the page timer and query logging. The page timer tells you how long it's taking for Drupal to build the page. There's also a query time which specifically address how many queries were run and how fast.
This will help you narrow things down. If your pages are built in 1s but it's taking 10s to get to the user's browser then there's something wrong with your server or perhaps with the size of your page. Query time will be a portion of the total page time, and the ratio of queries to ms should be queries > ms. For example, if it takes your server 1500ms to execute 500 queries then something is wrong.
WebOzy: the web services company
About Me
As a non-Drupal issue
I'd say right away that you need to revisit the sizes of the images your downloading with that page and resize them to bring down their size. It looks like the images themselves represent >1 MB of data that needs to be downloaded with the page load.
You might also want to see if there are ways to reduce the number of files that need to be downloaded. The fewer the files, the fewer the connections that need to be made. If you use webpagetest.org or tools.pingdom.com, you can see the number of files that are being downloaded and the size of each. Image files can be reduced by the use of image sprites. You might be able to condense .css and .js files together.
Finally, I recently moved my site from a shared-server environment to a dedicated server and that did a lot to improve page loading performance. If you're on a shared server you're competing for resources against other websites. Also, if you're content is static, you might consider a Drupal module like Boost, that will allow the page to be cached instead of being built each time the page request is made.
Good luck!
Also, looks like you're not
Also, looks like you're not really utilizing Varnish. Check out http://www.isvarnishworking.com/ for checking that. You said that you're on D6. Are you using Pressflow or patched core?
For all the images and such you should check out https://www.cloudflare.com/ I just rolled that out for all the clients that I manage. Just make sure that you update your Varnish config: https://www.cloudflare.com/wiki/Varnish_Cache
WebOzy: the web services company
About Me
Have you got a whitelist of
Have you got a whitelist of cookies in varnish that should not be served cache content e.g. the sessions cookie SESS?
If you're passing any request with any cookie at all immediately to the backend then none of your content will be cached for any users at all.
http://www.linkedin.com/in/linuxwizard
Thanks for all the advice
Thanks for all the advice guys. I'll checkout all of your suggestions and report back.
Not only images ...
Images may be part of the problem, but it is not the only problem.
Using command line, I see that it takes 5.8 seconds to get the first response from that URL, which is the Drupal backend processing time.
You have something else other than images causing this.
Drupal performance tuning, development, customization and consulting: 2bits.com, Inc..
Personal blog: Baheyeldin.com.
You seem to be correct that
You seem to be correct that varnish may not be configured properly. Unfortunately, I'm not sure the best way to fix this. Is it appropriate to have varnish ignore session cookies or should I be correcting the cache-control settings for various pages on the site? Most of them appear to be using no-cache. The pages I view do show up when I run the varnishtop command. I'm a bit of a novice with varnish.
Also, we're using Pressflow.
Here are my Devel
Here are my Devel results:
Executed 389 queries in 131.11 milliseconds. Page execution time was 2547.34 ms.
Firebug says my page loaded in 3s. Not terrible but not great either.
I would say the queries are not part of the issue.
For Varnish config, see
For Varnish config, see https://fourkitchens.atlassian.net/wiki/pages/viewpage.action?pageId=484... (change the version to see configs for D6 and older Varnish versions).
Query time looks OK, but that page execution is pretty nasty. Given how fast the queries are, the page execution time should be way faster. Could be APC (http://2bits.com/articles/importance-tuning-apc-sites-high-number-drupal...) or it could be module. It's likely a bad module or bad config.
WebOzy: the web services company
About Me
Do you have developed your
Do you have developed your own module(s) or so? Maybe something is wrong in there, e.g. hook_nodeapi() doing lots of heavy lifting?
We have quite a few custom
We have quite a few custom modules for on our site and 3 or 4 of them do implement hook_nodeapi. We also have masquerade installed which I have read can cause problems by setting the session cookie even for anonymous users. I applied a patch from http://drupal.org/node/705858#comment-3558688 to my dev site but the session cookie is still being generated for anonymous users. Any ideas on the best way to track down which module is causing this? Also, no-cache seems to be added to every page, even static pages, which doesn't make sense.
APC seems ok, 192M shm size, 46M free, 0% misses and 0.18% fragmentation.
I don't know of any other
I don't know of any other check for hook_nodeapi() then having a look at those manually.
hope that helps
For the session, stick this
For the session, stick this in a page template:
<?phpprint_r($_SESSION);
?>
and for the long page execution, you can try something like http://drupal.org/project/profiling/
WebOzy: the web services company
About Me
XHProf indicates _theme_process_registery
Still working on this page load issue. I ran XHProf on my local dev server (I'm the only user) and it indicated that the major hogs on the page load were _theme_process_registry and function_exists.
Even with only 1 user on my dev site I'm getting pretty terrible page load times.
Page execution time was 10413.43 ms. Executed 405 queries in 127.57 milliseconds.
Has anyone come across issues like this with _theme_process_registry?
theme registry and function_exists
There are a couple possibilities here. First, you may have "Rebuild theme registry for every page" turned on under your theme settings. This will cause a huge performance hit and is likely responsible for many/most of those function_exists calls. If that is not the case, then you may have a cache configuration problem and the theme registry is failing to be properly stored in cache.
Second, check out the module_implements cache patch (http://drupal.org/node/557542) to reduce the number of function_exists calls which occur when detecting hook implementations.
Monitor APC and make sure shm_size
Also remember to monitor your apc. If the apc shm_size is not big enough to hold the entire codebase, php will slow to a crawl. What happens is the APC cache has a max amount of space it can use. Once it hits this, then it will need to reset/clear the space and rebuild the cache all over again. If you're hitting this max a lot APC is adding a lot of overhead to your page load, rather than reducing.