Complex site, Slow Drupal - Is this normal?

The TTFB (Time to First Byte) is larger than normal = 0.738s

See you test result page here: http://www.webpagetest.org/result/130109_SS_78V/1/details/

A long TTFB usually is caused by slow responses via the database, and/or excessive modules (modules that effect the final html sent).

Also some of your css assets are a bit slow loading. This one in particular:

Request #6
Details Request Response
URL: https://www.see.asso.fr/sites/see.shareontheweb.com/files/advagg_css/css...
Host: www.see.asso.fr
IP: 87.98.176.200
Location: France
Error/Status Code: 200
Start Offset: 0.935 s
Initial Connection: 59 ms
SSL Negotiation: 87 ms
Time to First Byte: 122 ms
Content Download: 2007 ms
Bytes In (downloaded): 35.1 KB
Bytes Out (uploaded): 0.6 KB

2 seconds to load a 35.1Kb style-sheet?

The actual Start Render time (browser side) = 3.357s
Document Complete Time = 5.899s
Fully Loaded Time = 5.946s
Data Sent: 642 KB
DOM Elements: 881

The test was performed From: Paris, FR - using IE8 - DSL

Most of this points to a excess number of modules. I trust you are using APC - at least (for PHP cache benefits)?

Hard to believe that 400 modules are needed to achieve this website, even with solr, organic group, or i10n features. The custom modules code have to be checked too, maybe it has to be refactored for faster execution.

Massive amount of modules will be hurting you (you have a huge amount of function calls per page). From your xhprof runs it also looks like you are using Panels, but you dont have any panel caching enabled (see time spent in panels_render_display and other panels_ functions). You will need to go in and configure your panels displays to use caching, which should give you a significant boost.

You will also want to do the same for views.
For caching you'll want to make sure you have APC on, and i'd recommend also using either Memcache or Redis, which should also give you a boost.

Also, that is an awful lot of mysql queries you have per page. You will want to see if you can disable any modules you have enabled, and also turn on the devel module and output the query logs for the page. Examine any unecessary queries or where anything can be cached (to use cache_get).

Lots of time spent at drupal_lookup_path in http://xhprof.shareontheweb.com/?run=50ed0ffe87d9c&symbol=db_query&sort=...

A bit further gives that the special_menu_items module using loads of time: http://xhprof.shareontheweb.com/?run=50ed0ffe87d9c&symbol=call_user_func...

Have you considered splitting the site across several drupal installations or a multi site setup, for example the e-commerce side could sit in its own installation on a seperate subdomain.

Also on this page https://www.see.asso.fr/prix_distinctions. you are scaling images from very large sizes to thumbnail size, you could improve this significantly.

Speed test result:
https://www.see.asso.fr/sites/see.shareontheweb.com/files/editorimages/D... is resized in HTML or CSS from 1600x1071 to 130x87. Serving a scaled image could save 345.2KiB (99% reduction).

See the comment above on images, as they represent a significant portion of bandwith.

If you assume (reasonably) that each Drupal module loaded / accessed (on a page request) requires ~8Kb of memory (and cache space); the APC Cache total would need be 8Kb * 400 = 3,276,800 bytes.

So ideally, to minimize the impact of pulling raw disk data for 400 modules at the I/O disk level, the APC PHP cache would need to be increased to 3276800 bytes = 3.125 Gb. With likely APC fragmentation resulting, that should be more than doubled!

A reasonable conclusion, increase System memory and caches, or start pruning / optimizing 400 modules. An average Drupal module will typically add about 10 function calls to every GET (page) request.

If you are happy with the current anonymous I/O times, but looking for a sharper edge for 'logged in' users; then consider the use of Nginx's micro-caching. See: http://drupal.org/node/1408410

Thanks, I already have many things to look at...

• I am running mod_php, not fcgi - is that good?
• APC has 1G, but never asks for more than 300M
• e-Commerce on a separate install would negate most of the benefits of the current design. Any product sold is related to content relevant in the main site, grants access to roles, OG groups, add users important runtime information....
• how can I cache blocks? I already set the "cache_block" and "cache_menu" table in cacherouter (mysql tables are empty), the performance page does not allow the set block cache because of access control modules, but boost shoudl take care of everything.
• I just cached panels: only one option is available "simple cache" that I set for 30 minutes, argument granularity (because they load differently for each of the 100+ groups)

I have found significant improvements for large sites through the combination of APC and Varnish, https://www.varnish-cache.org/about.

Have you considered Varnish?

Including things other people have said, I would also switch to Pressflow,which is a high performance distribution of Drupal. It contains several core patches to improve performance. One that would greatly benefit you is the drupal_lookup_path cache. Drupal does not cache path lookups, but Pressflow does and will give you a significant decrease in the number of queries.

Other suggestions:

As others have mentioned, you got some large images that should be resized. If you're using mod_php (most likely), then serving this static content will eat up a lot of memory as PHP is still loaded by Apache when serving it.

There are a few ways to handle anonymous pages. The easiest for you is using APC. As others have mentioned, APC can become fragmented with large object caches, but Cacherouter does allow you to mix and match caching methods. Use APC for the page cache and cacherouter for everything else, but always keep your cache_form in the database!

A better solution would be to use Boost for anonymous pages. That way PHP doesn't have any processing to do, so long as a page is cached.

The best solution is to setup Varnish on the server and let that handle cached pages. Varnish also takes have some some pretty significant server load when dealing with static content.

On the database.

You have an awful lot of queries there. The time is good, but you might be having query cache mess with you some too. If you have a development copy of the site, disable MySQL's query cache and enable query output in the devel module and check the queries again. You might have some in there that really take a lot longer to generate and could use optimizing.

400 modules???

This really is a killer. I would re-evaluate these with surgical precision. Many times people use bigger modules but only use one small part of it. If you find this in your scenario, then I would move the functionality you need to a custom module and cut a bunch of the fat. I took over a site from a larger Drupal shop a couple of years ago that had over 160 modules and real slow loading times. I did this very thing and reduced them down to less than 100 with no loss in functionality and a performance increase more then 5 fold.

Is that the CSS Embedded Images module doing all those 200kb+ css files? That's working horribly. Every new page seems to have a new advagg css bundle and each has 200kb+ of inline images in it.

So rather than caching the images once, they're being downloaded over and over. And they're taking 2-10 seconds per page. And some of them are admin images that your users will never see like the Farbtastic color wheel. See https://www.see.asso.fr/sites/see.shareontheweb.com/files/advagg_css/css...

Check out item number 16 on
http://www.webpagetest.org/result/130109_AM_PKS/1/details/
It's 232.8 KB and takes 5.25seconds to load. And since it's css and not jpg/gif it's blocking the rendering of the page.

On the front page it's item 21:
http://www.webpagetest.org/result/130109_7X_PN6/1/details/
and it's a 234.1 KB advagg bundle with the exact same images embedded.

You might try disabling advagg and css_emimage and using stock drupal aggregation until you can get all that sorted out.

"Time to first byte" of 1.5 seconds might not be great, but the reason you're waiting 5-10 seconds to "start render" is all the front end stuff: css and js.

A different approach: put the static files (css, js, images) in another server running lighthttpd (http://www.lighttpd.net/). With it, you can reduce the answer time, the number of requests and, of course, the size of the files (if you check the CSS, you can reduce a lot just changing the path of the images).

I know, it's hard to do it because involve new server, new setup, etc etc etc but if you REALLY need, can be a good way to reduce time.

Before that, you can reduce the size of the HTML generated removing comments, spaces and everything that you don't need.

2 cents more: looking the webpagetest results, you're spending 174ms just to redirect the see.asso.fr to www.see.asso.fr. Try to check using the right URL (www....)

All the best

It probably only needs to be used on cart pages

so thats an easy win - cart* https everything else http

your changes thus far have improved the css, js & image download times

Using the CDN module, you'll be able to use a separate domain for static files, such as .js and .css, even images and other media. This has the advantage that your browser can open more connections simultaneously (because it's a setting per hostname), resulting in more content being downloaded at once (i.e. shorter 'waterfall' in the speed tests).

I've also done some improvements on my site by offloading some javascript from the header file to the footer (drupal_add_js has a 'scrope' option). This has the nice effect of page rendering being called earlier. The full load will not necessarily be shorter, but it will 'feel' faster.

Definitely looking better. I'm getting Boost pages here in New York, so the html is coming pretty quickly.

Good advice in here.

You could investigate why the two theme css, grid16-fluid.css and local.css, aren't being aggregated and fix it to save two requests, but they'll be cached, so that's a bit nitpicky.

As Martijn says, you could lighten your < HEAD > and initial render by moving the js. Either changing the scope of individual files or you can try the quick and easy way of moving the $scripts variable down to just before the $closure variable in page.tpl.php. This may screw up some javascript, but often it does not. The modern way is to use a script loader like http://drupal.org/project/labjs to load the scripts asynchronously. That can also mess with your javascript. The addtoany script is already loading asynchronously, so that's not a problem.

As a heavy nginx user, I'll add a +1 to putting the css, js, and images on alternate domains/subdomains using CDN and serve those from nginx. It will also save your server memory over using apache2.2+mod_php and make your server run better in general. However, installing and configuring nginx is not as easy as enabling a module.

Or you can put nginx in front of the whole thing and then you'd get the benefit of serving boost static files from nginx as well, and probably faster ssl negotiation, but that will take more config work than just doing the static files.

If I am reading correct - you are using APC as user cache????????????

1. STOP DOING THIS RIGHT NOW!!!! Because now it looks like APC is swapping disk.

2. Use APC only for what it is designed for: opcode cache, that means do with drupal apc related modules exactly nothing. Just look at phpinfo() to check if it is running and has in your case 128-1024M size

3. If your traffic is anonymous - Boost is the way to go. If it is mainly registered, then check views and block caching.

4. Entity cache module should be enabled in any case.

I cannot say more, because it very much depends on the configuration of the site, so the steps I have described are more or less safe on any site.

See http://www.memonic.com/user/2ni/id/1cefk for an explanation of APC fragmentation and how to fix it.

Are you using APC for cache_form? If so, then switch it to using the database. There's a couple of reasons for this. First, that cache can get huge and eat up a lot of memory. Second (and most important), if the cache is cleared while a user is filling out a form, then they will get a validation error when submitting the form.

A lot of times you can evaluate your site and end up going with native database caching. It depends on how hard your database is working. If you got some performance to spare, then database caching is faster than APC or memcache. Remember, MySQL also has a cache in it, the query cache, so tuning that and MySQL over all and just using that for cache can, a lot of times, make the site even faster. A lot depends on how write intensive your site is and how often cached objects have to be regenerated.

Well, I have another factor to add into the mix: the type of webserver <-> PHP interface.

I run a few servers that use Apache with FCGI (not FastCGI). So a few CGI processes are started and used by multiple requests. APC cache is NOT shared between processes in this case, meaning there are several cache instances running at the same time, and they can all (possibly) be caching different things, leading to odd results, especially if a user is not bound to a specific process. I've noticed this when refreshing apc.php a few times, and noticed the results differed a lot a few times.

Now I am using PHP-FPM, which does share the cache, and I think mod_php also does (but has its own drawbacks).

Just my two cents :)