file API

As announced, we've begun the Media Code Sprint to put better media handling into Drupal core!

It's been a great time so far. In the morning, drewish compiled a file function guessing game (free beer!), and andreiashu took the bait and completed the first round with great feedback about how unusable the current state is.

Next, we've sat down and begun writing SimpleTests for the hook_file patch. This has been great fun for me personally; though drewish is an old hand at building tests, this is completely new for me. And to top it off, webchick, Queen of Drupal testing, dropped by our table and got involved!

Even the process of writing tests has been helpful; we found at least one case for validation that had been missed in the original patch, and drewish decided that file_scan_directory needed refactoring. (He's currently chasing other problems as well, and cursing about finding himself going down rabbit holes.)

There's more fun to be had for all! If you want a quick 15 minute task that everyone is qualified to do, go play the File API Function Guessing Game. If you want to do more, then ping me (aaronwinborn), drewish, or dopry at #drupal in IRC, or leave a comment here!

(Cross-posted at AaronWinborn.com.)

The Media Code Sprint is underway! Here's a cross-post from my blog detailing the goals of this sprint, which runs through Saturday. We need your help!

Andrew Morton (drewish), Darrel O'Pry (dopry, remotely), and I are heading up a Media Code Sprint in Portland this week! Come help, in person or remotely, if you're interested in multimedia and Drupal! It has now officially started, and as I've volunteered to help keep folks updated, here goes...

First the reasons.

Number One: Better Media Handling in Core

Dries conducted a survey prior to his State of Drupal presentation at Boston Drupalcon 2008, and number one on the top ten (or 11) list of what would make THE KILLER DRUPAL 7 Release was "Better media handling".

Let me repeat that. Better media handling.

People have done really amazing stuff in contrib, but it is difficult (if not impossible in many cases) for developers to coordinate the use of files, as there is no good means for file handling in the core of Drupal. Thus, we have several dozen (or more) media modules doing some small part, or even duplicating functionality, sometimes out of necessity.

We need (better) media and file handling in Drupal core. In particular, there has been a patch for a hook_file in the queue for over a year, which has been in the Patch Spotlight (for the second time, no less) since May! (And has been RTBC several times during that process...) Come on folks.

One of the powers of Drupal is its system of hooks. We have hooks to modify nodes, to notify changes to user objects, to alter nearly any data (such as forms and menus). Noticeably absent is a consistent handling for files or any sort of notification. We need hook_file.

So goal Number One: get media handling in core. The means? Add hook_file and make files into a 1st class Drupal object. We'll be creating a test suite for functionality in the hook_file patch to validate it and "grease the wheels" to get it committed.

The other goals of this sprint pale in comparison to the first in utility, but are still highly desirable and worthwhile.

Number Two: Refactor File Functionality in Core

This article I wrote should appeal to people interested in high performance Drupal sites. Most of the performance gains are achieved by doing things with the Javascript and Drupal's File API.

http://wimleers.com/article/improving-drupals-page-loading-performance

You can expect another article soon, about the same topic, but then putting it to practice. I'll use the CDN integration module (of which I'm also the maintainer) and the included core patches to do it in just a few simple steps.

Hello, I've recently submitted this patch for the backup module which allows automated regular backups of a Drupal installation to Amazon S3. It currently uses Geoffrey Gaudreault's Amazon S3 PHP class, which requires the Crypt_HMAC and (hacked) HTTP_Request PEAR modules, as well as forcing the site developer to alter some variables within the code of the class. I'm hoping the File API will offer a much easier way to integrate the backup.module with Amazon S3.

Sorry, I would've posted this to the What do you want in a file API? thread, but unfortunately you can't attach files to comments.

These are some use cases that Shane Peery, one of our clients, wrote up for AMBuzz, an intranet site for a large media distribution company. They have some fairly complex needs, so I figured I'd post them here in the hopes they might inspire some of the folks working on this stuff, and also in case anyone had any bright implementation ideas. :)

I've also attached some general observations/comments he made regarding Drupal's file API as someone with experience in programming, but new to Drupal. Some of the stuff he mentions is relatively easy to sort out (for example, the security stuff is a matter of turning on private downloads and moving the files directory outside of public_html), but I think there are some interesting ideas here anyway.

I'm cross posting an issue here to let you guys know of this way:

Private and Public File Access

Banner.module isn't really guilty here. There are a lot more modules that do not work with private file access.

For those looking to have Drupal serve private files efficiently, I've posted a simple core patch that adds X-Sendfile support:
http://drupal.org/node/74472

The handling and publishing of files is one of Drupals weakest points. My site was largely based on me publishing mp3 files and code. I wanted to upload the files with FTP and publish them from an upload directory on the server. At that point I made my own "filepublishing module". And from that point of I started to ponder on what could be done with the way Drupal handles files. And now I have a rough idea of how I would like to do it.
I probably won't have the time to do any time soon, so I'm sharing it here to make shure the idea is not lost when I start to think about other things.

So, one of the big things with static file serving is that it requires an entire Drupal bootstrap, which sucks up a lot of system resources. We have work to do in Drupal/the File API to support some interesting scalability options, remote files, etc. etc.

But, I think there are some interesting models for doing highly scalable static file serving outside of Drupal (and Amazon's S3 is cool).