For those looking to have Drupal serve private files efficiently, I've posted a simple core patch that adds X-Sendfile support:
http://drupal.org/node/74472
X-Sendfile is a feature provided by modern web servers (including Apache2 and Lighttpd) that provides a significant performance benefit for sites that need to offer large files for download (audio or video, for instance), but also need to enforce access controls by application logic.
In Drupal's case, while a bootstrap is still needed to determine whether to serve a file to the user or not, X-Sendfile avoids unnecessarily tying up the PHP process for the duration of the transfer, which has clear performance and scalability implications.
More information about X-Sendfile:
- http://celebnamer.celebworld.ws/stuff/mod_xsendfile/
- http://wiki.rubyonrails.org/rails/pages/HowtoSendFilesFast
- http://blog.lighttpd.net/articles/2006/07/02/x-sendfile
- http://lighttpd.net/documentation/fastcgi.html
Thanks to advice from Boris, I've tried to make the patch generic enough so that we could, conceivably, in the future support capabilities such as redirecting file transfers to an FTP server or an Amazon S3-based URL using the same core mechanism. This ties into some ongoing work related to Darrel O'Pry's filesystem module.
Editor note: nice patch! please continue discussion in the issue above. I am disabling comments here.