Performance, Aggreation, Private download, Caching, Node acces... Need advice

We encourage users to post events happening in the community to the community events group on https://www.drupal.org.
leilyrken's picture

Hi all,

My Drupal website is running very slow and we just wake up to trying to optimize the performance. I read a lot of thing about performance and caching system before and my main reason to post here is that I'm a bit confuse so I need some advice to find the best way to increase the perfomance of my site.

Maybe my post will also help high perfomance beginner.

First my website is:
- around 130 modules enabled (50 custom modules + cck, views, ctools, panels + different front end and back en modules)
- around 30 views, 10 panels (page/mini panel, home page is a panel), maybe 100 blocks used, 30 different content-type
- private file system
- node access control on a fiew page (simple_access)
- currently mostly anonymous page visited but will change
- content is quite static (a few part a really dynamic)
- SSL activated with secure_page for some page accessible to loggued user
- On homepage 170+ http request ( 35 CSS, 25 JS, 80 img from CSS, 30 img)
- APC installed and configured
- Squid caching installed and configured
- Deflate installed
- 1 server (4gb ram) for apache and 1 server (2gb ram) for mysql DB
- 15 - 23 second to load the home page the first time for a new visitor (!)
- Size of the home page 426 KB (!)

We already start to work on the optimization, here what we are doing:
- Reduce the total number of module, it can be reduce to 100 I think but we cannot loose functionnality
- Enable cache for all possible views
- Optimisation of the MySQL config according to Mysqltuner.pl
- Changing the theme to make it lighter
- Check custom code to optimize queries and set cache when possible (drupal_cache_set/get)

Now after reading, searching, testing about the performance I have a lot of question:
- With private download method I cannot enable the CSS / JS optimization; For the same reason I cannot use modules like css_emimage, css_gzip, javascript_aggregator... How can I reduce the number HTTP request ? How can I reduce the size of my home page ?
I tried a hack in the core (http://drupal.org/node/572516#comment-2737908) that works well actually but then I had to change a lot of modules to be compatible and I'm thinking that's it's not a good solution at all. For all new modules or update i will have to hack them. A real pain.
- With simple_access enabled I cannot enable the block caching. I found that I could use the module blockcache_alter to do it. Do I need to configure my blocks one by one with blockcache_cache ? Will it be same as if I use drupal block caching function ? Is it better that I write a custom code to limit access to my private page (for a user only) to disable simple_access and then enable drupal block caching ?
- I will optimize the DB config according to Mysqltuner.pl, is it enough ?
- About more global caching module : What will be the difference with Boost (http://drupal.org/project/authcache) and AuthCache, can I combine them ? I use APC, will cacherouter (http://drupal.org/project/cacherouter) increase my perfornmance ? The all point of this question is that I'm a bit lost with all the the caching level : I cache my block/custom code/view, then drupal cache the pages, then squid cache my website. At which level all those module will works, are they really required ?
- After all that is it enough ? what else can I do ?

Thank you

Comments

Start with pressflow and 2bits.com

jcchapster's picture

There are quite a number of things you can do.

First, start with Pressflow. http://pressflow.org/
This is drupal optimized for caching.
If you read the docs, you will find references to Varnish (for caching), APC, memcache.

There are a lot of articles that are excellent in the information at: http://2bits.com/
Khalid has written most of them, and they are excellent. His articles will help steer you towards getting a lot of performance.

This isn't a quick and easy fix. But if you take the time to read and understand, you will learn how to find the problem areas and solve them. Personally, not being an expert, and using what I have learned, I have made even older hardware perform amazingly well.

Good luck!

I didn't know Pressflow

leilyrken's picture

I didn't know Pressflow, thanks an lot, I will check what they've done. Thanks
About 2bits.com I found it during my research and I'm working on it (there's a lot)

Squid

mikeytown2's picture

If you have Squid installed correctly then something like boost will not help; unless your using its logic to control cache expiration. You said your using a private filesystem; try the parallel module. I have several reports that it helps a lot when using private downloads.
http://drupal.org/project/parallel

On more thing, if your traffic is mostly anonymous then why use private downloads?

At the current moment most of

leilyrken's picture

At the current moment most of the traffic is anonymous because the website was launch a few month ago but during the registration process and following the workflows, users have to upload theior own document and keep it for them. So I need a private download system.
I will do a test with Parallel on this website.

You've got quite a nasty

gchaix's picture

You've got quite a nasty challenge here. A few recommendations:

  • I'd strongly suggest refactoring how you do the user uploads (possibly using CCK filefield and CCK field permissions instead of the core upload) and get away from private downloads. As long as you have it set to private, you're going to have no end to the performance problems.
  • 130 modules? Egads! Surely you can combine and/or reduce some of those - especially the 50 custom - modules. That is a phenomenal amount of work to be do on every single bootstrap.
  • 4G of RAM might not be enough with that many modules and a reverse proxy cache like Squid. How large is your APC cache and is it full? Same for Squid - how large is it and is it full? What's your Squid cache hitrate?
  • Pressflow - learn it, use it. It may help significantly, especially in bypassing the bootstrap for anon users. It's perfectly safe - I've switched live sites to/from Pressflow with no issues at all.
  • Varnish - you already have Squid set up, so this might not be as useful. That said, I am a die-hard supporter of the Pressflow-Varnish stack. It does wonders for performance. It might be worth looking at replacing Squid with Varnish, but it likely won't fix the majority of your problems. You'll likely see an improvement in performance over Squid, but we're talking differences in the tens or hundreds of milliseconds, not the tens of seconds you're seeing here.
  • Cookies - eradicate them for anon user sessions. For the most part, cookie = no cache. There are very few valid reasons for a cooking to be set on an anon user's session. Look at your modules and root out the ones that set cookies on anon users - especially ones that don't clear the cookies. Once a cookie is set on a session, you're probably getting little or no caching on that session even if the content itself is eminently cache-able. Commonly-used modules that do this include Masquerade and Hierarchical Select, but I am sure there are quite a few others.
  • Memcache - if you're not using it, start. If you are, check your config. I've had better luck with using the Memcache API directly instead of Cacherouter on high traffic sites. When configured to use APC, Cacherouter seems to have cache fragmentation problems in my experience.
  • CSS sprites for theme images - it's a bit of work, but it'll reduce the excessive number of HTTP requests.
  • DON'T HACK CORE. I see you tried. Stop before you slide down that slippery slope into madness. You'll only regret it.

You forgot something. 170

bennos's picture

You forgot something.

170 Request???? Damm
Activate the aggregation of CSS and JS Files.

They can't - they're using

gchaix's picture

They can't - they're using private files. Hence my suggestion to find a different way to handle the uploaded files - so they can turn on public downloads and therefore enable aggregation.

Thank you very much for this

leilyrken's picture

Thank you very much for this complete reply. We will prepare a plan to apply that.

  • I will see how to handle the private download using public file system : I found http://drupal.org/node/189239 , I will try a similar method.
  • For the cookies, what do you mean by " Look at your modules and root out the ones that set cookies on anon users - especially ones that don't clear the cookies." ?

Cookies

gchaix's picture

Modules like Hierarchical Select, Views, and Masquerade store data in the session cookie. Anonymous users normally don't have a cookie (a good thing caching-wise) unless a module triggers its creation. Hit your site with an anonymous user session and take a look at the cookies. If there's a SESS cookie set on a fresh anonymous connection, it's likely you have a module enabled somewhere that's stashing something in the session. As long as there are cookies on the session, any external caching like Varnish or Squid is essentially bypassed.

For example:

Views stores exposed filters in the session. So if you have views filters exposed to anonymous users, Drupal will store the filter config in the cookie. This is definitely as it should be - if the user has set a filter you don't want to send them a cached version of the page that was cached with different filter settings - but it also means a cookie is set on that anonymous user's session and they're not getting cached pages anywhere else either. You may need to have filters exposed to anonymous users for your use case but if they're not required, make sure they're not exposed.

Hierarchical Select is similar - it stores its selection information in the session variable. Once a user hits a page with a hierarchical select-enabled form, they have a cookie. No more caching.

Masquerade, unfortunately, sets a $_SESSION['masquerading'] variable on every session - even anonymous users (where it's set to 'null'). Because that exists, every user has a session cookie and therefore bypasses the cache. Unless you need Masquerade, disable it so it's not setting cache-killing cookies.

I'm not picking on Masquerade or Hierarchical Select - I use and like them both, they just have this one little bad habit. :-)

There are ways (at least with Varnish) to force it to cache content even if cookies are set, but it can be complicated and tricky. It's vastly better to eliminate the cookies in the first place rather than try to work around them.

-Greg

Pressflow and sessions

mshmsh5000's picture

With so many modules in play, you should do an audit of when and where session is assumed before switching to Pressflow. Unlike gchaix's experience, I have seen unintended side effects from dropping Pressflow into an existing D6 site. Any module that assumes it can store info for anonymous users in session will get confused.

That having been said, all problems of this sort are eminently solvable, and PF is absolutely worth using.

leilyrken's picture

Hello all,

I'm back on the optimisation of my website. I did several successful update :

  • Reduce front page size by 2 by changing the theme and the block used
  • Reduce the number of HTTP request from 170+ to 110+ (We are still in private file system. I will try to change it later)
  • Reduce the number of loaded module from 130 to 80 by cleaning so unused module and refactoring for the custom modules
  • Remove the anonymous session
  • Remove node access modules and replace by custom code to enable block caching
  • Optimize the query and code
  • Optimize MySQL database configuration (and move the MySQL server on the same server with Apache. We had a big delay between the two server...)
  • Tune APC configuration (as describe in http://2bits.com articles)
  • Installed Parrallel module (but the change are not really visible)

Then we discovered that with the new version of the site, Squid was slowing down everything. When I run AB test with our without Squid I got really better performance without Squid. Ab test from another server in the same country:
- squid + parrallel : 39 Requests per second
- only squid : 52 Requests per second
- only parrallel : 166 Requests per second
- nothing : 171 Requests per second
I guess our configuration is not correct but event after trying to configure Squid according to http://2bits.com/articles/increasing-drupals-speed-squid-caching-reverse... It didn't change so we removed Squid...

I did a quick test with JS aggregation and CSS aggreation (with CSS Embedded Images module) and it reduce to 59 request and is quite faster. So it will be my next task.

However even if the website is now much faster, it's not perfect yet. I'm still looking what could make my home page displayed in less than 5sec
(5.520s for a first view, 5.976s for a repeat view)

After the AB tests I checked using a tool like http://www.webpagetest.org :
-> It seems that image from image cache are not cached by the browser. Is it normal ? Is there something to do ?
-> They said that I could compress more the image of the theme and from imagecache. How could I do ? For the theme I used http://lyncd.com/2009/03/imgopt-lossless-optimize-png-jpeg/ and for imagecache I have no idea.
-> No ETag headers : Do I just modify my htaccess to add it ?

After the file system modification, my next target will be to do some tests with:
- Pressflow
- Memcache

There is two other thing that I don't understand and that could limit my performance: the Memory use by Drupal and my Apache configuration

  • About the memory, using Devel on our local Xen server we have 16Mb used and on our Prod server we have 70 Mb used. For the same page and same code.

    Local xen:
    Page execution time was 598.2 ms. Executed 249 queries in 64.07 milliseconds.
    Memory usage:
    Memory used at: devel_init()=0.49 MB, devel_shutdown()=16.76 MB.

    Prod:
    Page execution time was 345.62 ms. Executed 290 queries in 25.76 milliseconds.
    Memory usage:
    Memory used at: devel_init()=1.78 MB, devel_shutdown()=50.15 MB.

    It's the same code and almost the same database (smaller on the local). Do you have any idea of what could be the problem ? (or what I could do to find it)
    I was thinking that it could be my Apache configuration
  • I checked our apache config:

    KeepAlive On
    MaxKeepAliveRequests 100
    KeepAliveTimeout 50
    Timeout 300
    <IfModule mpm_prefork_module>
        StartServers          5
        MinSpareServers       5
        MaxSpareServers      10
        MaxClients           30
        MaxRequestsPerChild   0
    </IfModule>

    I search how I could optimize it. First i had to evaluate the MaxClient. My server is 4Gb RAM. MySQL use 1GB max, The system should use 300Mb. Following http://2bits.com/articles/tuning-the-apache-maxclients-parameter.html and the related link, I had to check the size of my Apache process:
    Using top command with only a few activity on the site I've got:
      PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND
    24143 www-data  20   0 33756 1396 1108 S    0  0.0   0:00.00 su
    24144 www-data  20   0 20416 2088 1352 S    0  0.1   0:00.00 sh
    31482 www-data  20   0  382m 113m  63m S    0  2.9   0:02.32 apache2
    31483 www-data  20   0  368m 101m  66m S    0  2.6   0:02.16 apache2
    31485 www-data  20   0  380m 115m  68m S    0  3.0   0:04.02 apache2
    31734 www-data  20   0  382m  87m  37m S    0  2.2   0:01.98 apache2
    31737 www-data  20   0  344m  40m  28m S    0  1.0   0:00.82 apache2
    31826 www-data  20   0  370m  97m  59m S    0  2.5   0:01.30 apache2
    31952 www-data  20   0  369m  96m  59m S    0  2.5   0:01.14 apache2
    31953 www-data  20   0  377m  89m  44m S    0  2.3   0:01.04 apache2
    31957 www-data  20   0  341m  18m  10m S    0  0.5   0:00.30 apache2
    32000 www-data  20   0  376m  88m  44m S    0  2.3   0:00.76 apache2
    32632 www-data  20   0 18852 1172  932 R    0  0.0   0:00.14 top

Using top command during a AB test I've got :

  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND
31482 www-data  20   0  377m 108m  63m S    7  2.8   0:01.72 apache2
31485 www-data  20   0  380m 115m  68m S    7  3.0   0:04.02 apache2
31737 www-data  20   0  344m  40m  28m S    7  1.0   0:00.82 apache2
31953 www-data  20   0  341m  18m  10m S    7  0.5   0:00.42 apache2
31956 www-data  20   0  342m  19m  10m S    7  0.5   0:00.30 apache2
31158 www-data  20   0  378m 117m  71m S    6  3.0   0:02.46 apache2
31483 www-data  20   0  368m 101m  66m S    6  2.6   0:02.14 apache2
31732 www-data  20   0  368m  71m  35m S    6  1.8   0:01.48 apache2
31734 www-data  20   0  382m  87m  37m S    6  2.2   0:01.98 apache2
31826 www-data  20   0  344m  61m  49m S    6  1.6   0:00.86 apache2
31949 www-data  20   0  341m  18m  10m S    6  0.5   0:00.52 apache2
31952 www-data  20   0  341m  18m  10m S    6  0.5   0:00.40 apache2
31957 www-data  20   0  341m  18m  10m S    6  0.5   0:00.30 apache2
31958 www-data  20   0  341m  18m  10m S    6  0.5   0:00.28 apache2
31966 www-data  20   0  341m  18m  10m S    6  0.5   0:00.18 apache2
31967 www-data  20   0  341m  18m  10m S    6  0.5   0:00.20 apache2
31968 www-data  20   0  342m  19m  10m S    6  0.5   0:00.20 apache2
31969 www-data  20   0  341m  19m  10m S    6  0.5   0:00.18 apache2
31970 www-data  20   0  342m  19m  10m S    6  0.5   0:00.20 apache2
31971 www-data  20   0  341m  18m  10m S    6  0.5   0:00.18 apache2
31972 www-data  20   0  341m  18m  10m S    6  0.5   0:00.18 apache2
31731 www-data  20   0  355m  82m  59m S    5  2.1   0:01.38 apache2
31735 www-data  20   0  367m  94m  59m S    5  2.4   0:01.34 apache2
31959 www-data  20   0  341m  18m  10m S    5  0.5   0:00.26 apache2
31973 www-data  20   0  341m  18m  10m S    5  0.5   0:00.16 apache2
31997 www-data  20   0  341m  18m  10m S    4  0.5   0:00.12 apache2
31998 www-data  20   0  341m  18m  10m S    4  0.5   0:00.12 apache2
31999 www-data  20   0  341m  18m  10m S    4  0.5   0:00.12 apache2
32000 www-data  20   0  341m  18m  10m S    4  0.5   0:00.12 apache2
32001 www-data  20   0  341m  18m  10m S    4  0.5   0:00.12 apache2
24143 www-data  20   0 33756 1396 1108 S    0  0.0   0:00.00 su
24144 www-data  20   0 20416 2088 1352 S    0  0.1   0:00.00 sh
31944 www-data  20   0 18852 1180  932 R    0  0.0   0:00.00 top

In the first case, my process apache2 use around 95Mb, in the second case most of the apache2 process use 18Mb. What should I use to calculate my maxClient and what are your advice for the apache configuration ?

As you noticed I'm not a system admin, but I learn a lot everyday from here and I want to thank you all.