Best Practices for Scaling Drupal GRACEFULLY

Hi Folks,

It was great seeing everyone last night. I hope you all had a good time.

I wanted to pick your brains about a topic I know we all love - scaling Drupal!

In my spare time, I have been putting together a Drupal distro I hope to use for client projects, as well as a way to learn best practices for building sites. I am trying to build the distro with the assumption that at any time a client could get a huge spike in traffic, so the site should scale gracefully at any given moments without, or very few, hiccups. My challenge has really been seeing past all the caching options out there to really hammering down a strategy for scaling that would allow for these sites to handle thousands of visits per minute without us throwing new servers at it, clustering MySQL, etc.

As a note, none of these projects would receive millions of hits per second/minute/hour. I would expect that at most a site's would spike up to a low 10's of thousands per hour on the very, very high end.

So, the general practice I have taken to scaling is:

1) Use Pressflow instead of core Drupal.

2) Use the Boost module instead of Drupal core caching. I have run into issues on projects where they conflict, so I have defaulted to solely using Boost on all sites. Overall, I have seen better performance with Boost than with core caching.

3) Use Memcache to improve performance for logged in users. I have personally not done benchmark tests on this, but I have read a lot of articles and posts that allude to significant performance increases for logged in users when using Memcache.

4) Use Path Cache to minimize alias queries. I use path aliasing pretty heavily on sites, so I figured this would add some performance improvements.

There are some other strategies that were recommended to me that I am hoping you all can help demystify:

1) Using Cache Router as a replacement for Memcache and Path Cache.

2) Use Nginx instead of Apache as the webserver

3) Using Varnish or Project Mercury

There seems to be 20 different ways of doing the same thing within Drupal, but which is the best approach? What strategies have you all used to address scalability and what were some of the pain points?