Another file API conception

We encourage users to post events happening in the community to the community events group on https://www.drupal.org.
kkaefer's picture

Some people suggested that every module should handle files by itself. I am for a central storage of file information, though. This has several advantages such as being able to get a quick overview on all files, re-use files elsewhere and so on.

My suggestions include:

  • A single file handling mechanism
  • Multiple versions of one file (e.g. different resolutions, formats) and a hook that lets modules handle those versions
  • Revisions of one file (and if a file is changed, the above mentioned hook is fired that lets modules generate new versions from the file)
  • upload.module:
    • File uploading is changed to a file selection where you can select any of the available files or choose to upload a new one.
    • The file browsing widget allows users to search for files
    • Files (not file versions!) are attached to a node; modules can handle it how they want to (generate a slideshow, use them for inline pictures, provide different audio formats, different video formats, etc.)
  • Tables: files, file_revisions, file_versions and node_file for the mapping, term_file for taxonomies
  • node_file has a weight column so that file uploads can be rearranged
  • Provide a single location where you can manage all files, see to which nodes it's attached to and so on
  • Provide a function that lets other modules query the database for specific files (mime types, versions and so on)
  • Some features for the interaction with views.module
  • Let the user select where to store the data by using 'protocol handlers'. That means, the user can select if he wants to store it locally in a folder, transfer the files via ftp to another (file) server, use WebServices to store them and so on.

Example for the interaction between (a made up) image module and the files API:

  • User uploads file, file gets saved somewhere
  • hook_file_handle is fired, image.module checks if the file is an image and creates a new file version (thumbnail) with a different size
  • Modules can now rely on the fact that there is a thumbnail for each image and display that everywhere they want to
  • image.module can also check if there are more file versions (e.g. with different sizes) and generate those as well
  • If the file is changed (either in administer --> files or on the node's form), meaning a new revision is created, the hook is called again and the module can create a new thumbnail

Example for the interaction of img_assist.module with this system:

  • Users can upload or select files to attach to the node right on the editing form
  • Now they can insert the image with the right dimensions. If the user specifies any other dimension, the image is recalculated, a new file version is created

I'd love to hear some feedback on my conceptions.

Comments

Hurray.

dopry's picture

We're on about the same page. These are my goals for my filesystem.module. More eloquently stated. I'm still waiting to hear Walkah's plans. Do you have any time to do development on it? Could you take a stab at mocking up the UI elements?

.darrel.

nice writeup

moshe weitzman's picture

sounds terrific. protocol handlers could get complicated i think consider the impacts on a quote system or on creating hyperlinks. i would save that for later.

Yes, that’s probably the

kkaefer's picture

Yes, that's probably the last thing we'll have to implement.

chmod after upload

fen's picture

We've got our vhost security pretty tight, such that only the Apache process has R/W access to files. But uploaded files need to be (say) mode 0644 (everyone has read access) so that files can be downloaded.

A simple config setting that sets the default mode of uploaded files would be really helpful. Or I suppose this function could go into hook_file

All sounds good. I'll add...

sammys's picture

Great ideas and pretty much inline with what i'm thinking. Uploaded files require centralised management (FOR SURE!) and the ability for each file to be placed in any number of places. In addition to this, because the files are centrally managed, we'll require a reference count of each file so deletions can be prevented if the files are referenced anywhere. In this respect, some link between this and themes would be wise. Site logos and shortcut icons should be counted as site files.

As for the implementation of this. Count me on board to help out. This is the area I like to dabble in. :)

--
Sammy Spets
Synerger
http://synerger.com

I wrote up my own file

Dave Cohen's picture

I wrote up my own file handler ideas, which I'd like feedback on as well.

Thanks for sharing your good ideas. Raised one flag for me:

I don't like your idea of term_file for taxonomies. Personally I prefer the category module. Don't assume that everyone has taxonomy enabled. I feel like each file should belong to a node, and the node has taxonomy already. Why does the file need its own taxonomy?

Yeah, you’re probably

kkaefer's picture

Yeah, you're probably right about the taxonomy thing, that should be moved to taxonomy.module. But don't you think that making each file a node eats a lot of resources? That would introduce yet another node type which seems pretty useless to me.

Whoa! Yes! Each file a node

Dave Cohen's picture

Whoa! Yes! Each file a node == bad idea. I did not say that. I said each file should be affiliated with a node (and probably only one node). Each node could have any number of files, though.

I could be wrong. It might make sense for files to exist independently of nodes, the way users and taxonomy terms do. But I think each file belonging to a node is easy to understand and isn't limiting.

Some nodes may choose to have a 1-to-1 file to node mapping. That's up to the node.

I still don’t see the

kkaefer's picture

I still don't see the point why it makes sense to let a file only belong to one single node. Could you explain the advantages a bit? I am basically for a "file pool" where you just pop in your files and select which files are attached to what node. Modules cann now use the files attached to a node and display them in any given way. I often have files (mostly images) that should be attached to more than one nodes. But atm, I can't attach a single file to more than one nodes. If I update a file, I have to change all the files which is quite annoying.

Well, I did say I could be

Dave Cohen's picture

Well, I did say I could be wrong. However, I see this as raising questions and UI issues rather than solving problems.

It sounds like you want files to be like taxonomy terms. You'd create an interface for managing them in admin/files. A file pool is like a vocabulary. When editing a node (or user, or whatever), you could select one or more items from one or more file pools. Is that right?

In my experience, I've had to create node types with fixed file requirements. Say, a lesson node which consists of one video, one image, and three flash files. Always the same structure. When creating/editing that node forms prompt the user for exactly what is needed, and validate. In this case it makes sense for files to be part of the node, and each file can be identified by its node and label ('flash1', flash2', etc). In the only model where like taxonomy, I'd essentially create vocabularies for each node and the whole process would drive me mad.

So to address your file pool problem, I'd make a node type called file pool, and associate lots of files with each node instance. To modify the files, you'd edit the file pool node. Then other nodes (or users or whatever) can refer to those files and display the files mixed with their own content.

I guess its clear to me that some files should be always associated with a node. And if that's the case, why not all of them? It solves the problem of who can edit/delete the files (answer: anyone who can edit/delete the node that owns the file). Perhaps its simply an implementation choice. To me, it makes sense to implement files as part of nodes. It doesn't feel limiting at all.

File(s) as Node

sun's picture

Reminds me of: "Why should user profiles suddenly be nodes?"

If each file is a node (and multiple revisions of a file are the same node) you can leverage any existing Drupal functionality for nodes. You can vote on files, use access systems, assign them to groups, have your favorite node files and so on (the list is endless)... ay, forgot to mention the most interesting: With upcoming relationship API and upcoming CRE on your site you'll be able to suggest files to users based on their interests (=> knowledge management).

Daniel F. Kudwien
unleashed mind

Daniel F. Kudwien
netzstrategen

files

sun's picture

Just wanted to note that E-Commerce might rename its file module to ec_file, so that a overall fitting file API can take up this name. If you'll implement files as nodes :) I'd gonna vote +1 for taking "files" for your new project name.

Daniel F. Kudwien
unleashed mind

Daniel F. Kudwien
netzstrategen

Sounds great

poppaluv's picture

This solution sounds great. I've always had a problem with the way drupal handles files and it sounds like this model would be very flexible, addressing all concerns I've had, and many I haven't considered.

subscribing

geek-merlin's picture

subscribing

File API

Group organizers

Group notifications

This group offers an RSS feed. Or subscribe to these personalized, sitewide feeds:

Hot content this week