scalability

For those looking to have Drupal serve private files efficiently, I've posted a simple core patch that adds X-Sendfile support:
http://drupal.org/node/74472

This group is an attempt to get a serious discussion started around database scalability with Drupal sites.

Right now, Drupal mostly relies on direct connections with mysql to generate Web pages. There is an internal caching mechanism which can help alleviate load, but which introduces some problems of its own (like the user login problem, where users have to reload a page after logging into the system). Typically, mysql optimizations and hardware concerns become a primary area of focus after Drupal has reached its caching limits. While there are not that many massively trafficed Drupal sites around (massive means greater than 100k hits an hour), they are coming, and it would be great to have some defined paths people could follow to build up their sites.

So, one of the big things with static file serving is that it requires an entire Drupal bootstrap, which sucks up a lot of system resources. We have work to do in Drupal/the File API to support some interesting scalability options, remote files, etc. etc.

But, I think there are some interesting models for doing highly scalable static file serving outside of Drupal (and Amazon's S3 is cool).