Handling Incoming Traffic from High-Volume Sites

I really found this: http://groups.drupal.org/node/5281 helpful when I first began exploring the Drupal forum on newspapers.

After reading your post on traffic problems, which was very detailed [thank you], it made me wonder why there wasn't a reference page created relating to server suggestions for developers considering Drupal for a newspaper site.

If there is such a page, I recommend the group admin add it as a link at the front of the forum, just like the link above is provided.

I am not a developer, but a publisher, and as a person who writes paychecks... I expect not only stories that draw readership (often through the web's power of linking), but a server foundation designed from the ground up in anticipation of building traffic.

It's great to see a story now and then explode w/traffic but finding work arounds that strip out all of the revenue [ads] from a page just so people can read it would make me furious.

While some readers would say...'THIS forum is about DRUPAL, not servers", I say, if your going to use a CMS then look at issues surrounding its limitations as it relates to the server or servers themselves. Afterall, servers are what makes the CMS funciton in the first place and with Drupal and all of its modules taxing a server's memory, its like using a truck that gets great fuel milage but can't carry the weight you haul unless the suspension is beefed up ahead of time.

I would like some feedback on this or helpful links to non-drupal forums that deal with server setup and configuration issues relating to traffic capabilities.

Thanks for the feedback! I talked to a former Web Editor, who basically broke it down to this: "Apache simply cannot handle massive volumes of concurrent connections in a low memory environment such as the sun's VPS." He pointed us to some resources about either replacing apache with nginx (http://drupal.org/node/110224) or running it in front of apache (http://groups.drupal.org/node/50168#comments).

While I like ideas like nginx and Boost, my main concern is that if something does go wrong in setting them up, something I can't discount (the list of modules for newspapers does have an explicit warning about Boost), then that would affect the website 100% of the time. Also, for those who don't have the server expertise for their paper (not out of question, given the current financial state of journalism), they may want to find a slightly different route.

In terms of Boost, too, if the issue really does come down to the number of HTTP requests, then I don't know how much it helps to serve a static page if it includes lots of images and Javascript which will also have to be retrieved with additional HTTP requests. The box to print/email/share our content alone has 8 small icons, each of which requires an HTTP request to retrieve afte the page is loaded.

While I've heard Amazon's cloud computing solution, EC2, is quite expensive, and I don't know if we'd want to try that transition anyway, one interesting idea someone mentioned to me was to use their S3 storage service not as a solution for all our images, but for some common images (e.g.: the masthead, the print/email/share icons, etc.).

One idea I've had, which is targeted at the 1% of the time we get hits, and requires coding but not server expertise, is to create some special templates file for a light version of the node page in the theme, which generates something similar to the stripped-down static page you saw. There could then be a module which could then enable this light theme for a specific node. The way I would do it would be to have template.php use the lightweight template files if it detected a node for which the lightweight version was enabled. (Hope that explanation makes sense.)

But in any case, I feel like the only way to get a good feel for what will and will not work would be to do some stress testing on our dev server. If anyone has any good ideas on how to do that, feel free to comment.

Solutions like fronting the system with NGINX fix a small problem (NGINX is faster than Apache for serving static files) while ignoring the big problem (a dynamic page generatiion cycle for Drupal, PHP and MySQL is hundreds of times more expensive). You need to cache generated pages outside of Drupal. That's what Boost does.

The cited warning about Boost refers to our experience over a year ago with a 5.x version of Boost that did not maintain the cache directory well. That's no longer the case; in fact, the current 6.x version does a very good job of keeping track of what needs to be purged.

If you're concerned about it, you can set the TTL on cached pages to something low, like 5 minutes, and still be VERY well protected against crushing traffic spikes from a Drudge or Fark link.

Keep in mind that Boost pages are served ONLY to anonymous traffic.

Flattened HTML pages (such as Boost creates) are served by Apache without being parsed for PHP, and of course there is no connecting with the database, running queries, et cetera. Unless you've done an extraordinary job of tuning your system, there will be an immediate and possibly stunning performance improvement for casual visitors. That's a bonus on top of the safeguard against Slashdotting.

I use Boost on my personal site, but we do not use it for our newspaper sites. We have Squid configured as a caching reverse proxy, connecting through a load balancer to a farm of multiple app servers running Linux, Apache, MySQL, PHP, Drupal and Memcache. In this scenario, Squid plays the same role as Boost, maintaining pages in a cache and serving them to unauthenticated visitors. It's a nontrivial thing to set up and maintain, and you don't need it for the typical smaller site.