File API

I've been doing some thinking about the future of the asset module. First of all, I'm completely dedicated to bringing it to Drupal 6 and beyond. What is not sure however is in which form it will move forward. With the improved hook_form and specifically with filefield and emfield tying into the youtube api, flvmediaplayer, the new media player et al, I certainly don't want to duplicate efforts. The asset module itself is quite a big project and maintaining it requires a lot of my time.

That's why I've been thinking lately of stripping the asset module down to the asset wizard: leaving the uploading of files to the *field modules (having the admin decide which modules he prefers and what type of files he needs) and managing them with the asset wizard. The asset wizard could be taken so much further if it could get all the attention it deserves. One thing I could use help on is the interface from a usability perspective: both on the asset wizard itself and on how to tie it in the different fields (filefield, emfield, ...) and editors (TinyMCE, WYSIWYG, ...). Everything is welcome: from concepts and wireframes to mock-ups and patches.

How do you want to handle the files and embedded media that have been already been imported by media mover, uploaded by filefield or integrated by emfield?

As announced, we've begun the Media Code Sprint to put better media handling into Drupal core!

It's been a great time so far. In the morning, drewish compiled a file function guessing game (free beer!), and andreiashu took the bait and completed the first round with great feedback about how unusable the current state is.

Next, we've sat down and begun writing SimpleTests for the hook_file patch. This has been great fun for me personally; though drewish is an old hand at building tests, this is completely new for me. And to top it off, webchick, Queen of Drupal testing, dropped by our table and got involved!

Even the process of writing tests has been helpful; we found at least one case for validation that had been missed in the original patch, and drewish decided that file_scan_directory needed refactoring. (He's currently chasing other problems as well, and cursing about finding himself going down rabbit holes.)

There's more fun to be had for all! If you want a quick 15 minute task that everyone is qualified to do, go play the File API Function Guessing Game. If you want to do more, then ping me (aaronwinborn), drewish, or dopry at #drupal in IRC, or leave a comment here!

(Cross-posted at AaronWinborn.com.)

We're running SimpleTests on various core modules, to get a baseline of what is fine, and what breaks with the hook_file patch.

I'm going to comment to this thread with results. The first run is on a clean installation of the build of Drupal 7.x-dev, Last updated: July 24, 2008 - 08:03.

Tests to run:

blog api
simpletest
system
upload
user
cache

As part of the Media/Files Code Sprint we're trying to re-evaluate the mess of badly named functions in file.inc. When we were discussing it on IRC, clouseau pointed out that it's been a popular drinking game at DrupalCon for a few years.

Take a look at each item in the list and make a guess about what you think each function does based solely on the name--to make this a fair fight I'm even giving you the parameter names--then look at the documentation and see how you did.

Please leave comments telling us which were the easiest to guess and which functions totally puzzled you. This is a big list so feel free to comment even if you don't make it all the way through.

The Media Code Sprint is underway! Here's a cross-post from my blog detailing the goals of this sprint, which runs through Saturday. We need your help!

Andrew Morton (drewish), Darrel O'Pry (dopry, remotely), and I are heading up a Media Code Sprint in Portland this week! Come help, in person or remotely, if you're interested in multimedia and Drupal! It has now officially started, and as I've volunteered to help keep folks updated, here goes...

First the reasons.

Number One: Better Media Handling in Core

Dries conducted a survey prior to his State of Drupal presentation at Boston Drupalcon 2008, and number one on the top ten (or 11) list of what would make THE KILLER DRUPAL 7 Release was "Better media handling".

Let me repeat that. Better media handling.

People have done really amazing stuff in contrib, but it is difficult (if not impossible in many cases) for developers to coordinate the use of files, as there is no good means for file handling in the core of Drupal. Thus, we have several dozen (or more) media modules doing some small part, or even duplicating functionality, sometimes out of necessity.

We need (better) media and file handling in Drupal core. In particular, there has been a patch for a hook_file in the queue for over a year, which has been in the Patch Spotlight (for the second time, no less) since May! (And has been RTBC several times during that process...) Come on folks.

One of the powers of Drupal is its system of hooks. We have hooks to modify nodes, to notify changes to user objects, to alter nearly any data (such as forms and menus). Noticeably absent is a consistent handling for files or any sort of notification. We need hook_file.

So goal Number One: get media handling in core. The means? Add hook_file and make files into a 1st class Drupal object. We'll be creating a test suite for functionality in the hook_file patch to validate it and "grease the wheels" to get it committed.

The other goals of this sprint pale in comparison to the first in utility, but are still highly desirable and worthwhile.

Number Two: Refactor File Functionality in Core

Hi,

Looking for suggestions for D5 non-node file handling, perhaps someone has done this before, perhaps there is a useful helper module.

So, basically I need my module to manage files without having a nid to attach it to. The files table has a nid column. Should I just manage my files separately from the files table?

Cheers
Simon

I'm declaring the week of OSCON as a media and files code sprint in Portland Oregon.

I would like to take an opportunity and introduce yet another file framework for Drupal 6. The project page is http://drupal.org/project/fileframework.

In order to draw more attention and reviews to patches that would improve the file support in Drupal 7 I've started the following list.

#247095 upload.module hard-codes 'view uploaded files' permission check
#203204 Uploaded Files have 600 permissions
#142995 Add hook_file and make files into a 1st class Drupal object

This article I wrote should appeal to people interested in high performance Drupal sites. Most of the performance gains are achieved by doing things with the Javascript and Drupal's File API.

http://wimleers.com/article/improving-drupals-page-loading-performance

You can expect another article soon, about the same topic, but then putting it to practice. I'll use the CDN integration module (of which I'm also the maintainer) and the included core patches to do it in just a few simple steps.

This week I "got around to it". That thing I knew I should do for so long, but never actually wanted to do. Well this week the itch could not be ignored. Finally I scratched. And now I share with you...

UpAPI

I thought of calling it "Yet Still One More File Helper Module," but decided shorter was better. Because shorter is what this module will do for you, if you need to add file uploads to your module, that is. Your code will be shorter and your time spent will be shorter. At least that's the goal, and you can help by playing with this new module and giving some feedback.

Please help test the fine patch at http://drupal.org/node/115267

Application for Summer of Code 2007: MultiPublisher Module

by Ryan Grace www.thefractal.org :: ryan@thefractal.org

Synopsis

Greetings! I don't intend for this to be an advertisement for Amazon, so if folks want to discuss other similar services that's totally cool, and we might even change the name of the group. But, I've been spending (too many) hours researching Amazon Web Services recently and thought some people might be interested to know what it's all about and how they might be able to use this stuff...

Hello, I've recently submitted this patch for the backup module which allows automated regular backups of a Drupal installation to Amazon S3. It currently uses Geoffrey Gaudreault's Amazon S3 PHP class, which requires the Crypt_HMAC and (hacked) HTTP_Request PEAR modules, as well as forcing the site developer to alter some variables within the code of the class. I'm hoping the File API will offer a much easier way to integrate the backup.module with Amazon S3.

So there is quite a bit of duplicated code and convoluted interfaces for manipulating files. especially images...

imagecache, image, watermark, imageexact, etc, etc are all relatively simple 'filters' or 'processors' for files. I've got different versions of imagecache with various gd filters, image2ascii conversion, mosaic affects, etc floating around, along with some tools for processing quicktime files and front-ends for ffmpeg I'd like to make available to the community, but coding a module for each one would be a real pain, and would make for a rather module heavy drupal...

imagecache is really a proof of concept work for generating images on the fly, but I had to throw the 'preset' and 'action' management interface into it so it could do its good work. I think there is a strong need to sort this kind of processing into its own little library. Actions.module or filter.module would be a good model for a UI I think.

Sorry, I would've posted this to the What do you want in a file API? thread, but unfortunately you can't attach files to comments.

These are some use cases that Shane Peery, one of our clients, wrote up for AMBuzz, an intranet site for a large media distribution company. They have some fairly complex needs, so I figured I'd post them here in the hopes they might inspire some of the folks working on this stuff, and also in case anyone had any bright implementation ideas. :)

I've also attached some general observations/comments he made regarding Drupal's file API as someone with experience in programming, but new to Drupal. Some of the stuff he mentions is relatively easy to sort out (for example, the security stuff is a matter of turning on private downloads and moving the files directory outside of public_html), but I think there are some interesting ideas here anyway.

I'm cross posting an issue here to let you guys know of this way:

Private and Public File Access

Banner.module isn't really guilty here. There are a lot more modules that do not work with private file access.

Hey there - I work for the Electronic Frontier Foundation (EFF), and we are migrating our huge old legacy site into Drupal. We are trying to get some help developing some additions to the file upload module, and Zack suggested I post our specs here for feedback from you all.

Here's the email I sent him with the specs of our dream file upload module:

EFF needs a file upload and organization module that interacts with files stored on the filesystem. There are some contributed modules with similar functionality, but none that do exactly what we need. It's my belief that adding functionality to the existing 'upload' module that comes with Drupal would be a fine approach.

The handling and publishing of files is one of Drupals weakest points. My site was largely based on me publishing mp3 files and code. I wanted to upload the files with FTP and publish them from an upload directory on the server. At that point I made my own "filepublishing module". And from that point of I started to ponder on what could be done with the way Drupal handles files. And now I have a rough idea of how I would like to do it.
I probably won't have the time to do any time soon, so I'm sharing it here to make shure the idea is not lost when I start to think about other things.

In another comment (Multisite with separate files/ per vhost) I discuss a problem that I am having with moving databases between multisite installations when they each have their own files/ directory. It concludes:

A solution is to not store the filepath prepended to every filename in files.filepath, but rather to prepend the filepath when needed. Not sure where to begin (maybe file_create_path) but I thought I'd float the idea to see what others think of it before I actually attempt to patch core - and probably some modules...

My primary concerns regarding file handling are

• Ensuring only authorized users can download files
• Making it easy to add a file field to a node form

I've posted more file handling ideas here. Please take a look. I'd like feedback.

Some people suggested that every module should handle files by itself. I am for a central storage of file information, though. This has several advantages such as being able to get a quick overview on all files, re-use files elsewhere and so on.

So, one of the big things with static file serving is that it requires an entire Drupal bootstrap, which sucks up a lot of system resources. We have work to do in Drupal/the File API to support some interesting scalability options, remote files, etc. etc.

But, I think there are some interesting models for doing highly scalable static file serving outside of Drupal (and Amazon's S3 is cool).

I was hoping to have some of these questions answered in the poll, but most people have chosen to just cast a vote and not comment.

What dopry wants...
-- specific processing based on rulesets(mimetype, extension, nodetype, userroles, etc).
-- queued processing for long time functions (ex. transcoding media files)...
-- finegrained access control.
-- themable rendering.

I have other stuff kicking the back of my head, but its mostly architectrual stuff to make it more accessible.

There is a horrible line between file.inc and upload.module. File.inc supposed provides hooks like hook_file_download but these get hijacked by upload.module. Also, file.inc does not define or use any database tables (which is great, as using a database for files should remain optional imho), but upload.module does and confusingly names the database table 'files' rather than 'uploads'.

Its important that we make a definite separation between the core file.inc file and the optional upload.module.

• File.inc should provide hooks and functions for use with files which should not restrict people to having to make a file a node or store any details of a file in the database.