Media/Files Code Sprint in Portland, Oregon

drewish's picture
Start: 
2008-07-23 (All day) - 2008-07-25 (All day) US/Pacific
Event type: 
Drupalcamp or Regional Summit

I'm declaring the week of OSCON as a media and files code sprint in Portland Oregon.

Since aaronwinborn is getting into town on Wednesday and I've got pay work to do I'm bumping the schedule to start it on Wednesday. I'll be doing "regular" business hours at Tiny's Coffee Shop which is near the convention center, if you're in town you should stop by and hang out. dopry has said he'll be available to help out via IRC.

Comments

Eh?

ZaphMann's picture

Good for you, although I have no idea what you are talking about!

Listen to Andy Partridge (XTC) for a better life.

Listen to Andy Partridge for a better life.

...and if you read, buy a book

I would love to attend...

arthurf's picture

...but dates totally stink for me. I'd love to do a media files summit and work on integration between some of the major media modules. Any chance you'd consider the following week?

possibly

drewish's picture

i'd been tossing those dates around for a bit but i'm not married to them. my only concern would be that moving it to the next weekend might be bad for other people...

We could be up for this

bonobo's picture

We'd be up for working on this, and even more so if this moves forward -- one of the things we've been looking at (for a pretty long time, really) is a more uniform way of treating/handling files. We have bumped up against this on a few different jobs now.

Cheers,

Bill


FunnyMonkey
Tools for Teachers

re: Agenda...

dopry's picture

I'll be there will bells on, and my own agenda...

My primary focus for this sprint in order of important ImageAPI 2.x, File Form Elements, and File Access Control. (No not public & private files, but access control for private files).

File Management Needs

eliza411's picture

drewish, here's the file management write up that I promised to send to you. I realized I don't have your address.


Our client currently has three 'styles' of files which they upload to the Web site:

Files as primary content:
This is highlighted on the Staff Forms page, where the downloadable document is the thing a user really wants. We're currently uploading these with the file field of CCK and using an Upload File content type to categorize the documents by function, department, and audience and to automate their display on site-wide or (eventually) department-specific pages with no further action on the part of the user.

Some authors appreciate the feature; others chafe at the discipline of providing the guidance to users.

Files as secondary content:
This is the meetings and minutes style content, where the primary information is the date and location of the meeting, to which the files are attached as they become available. This is accomplished with the CCK file field.

Having a special field for each type of file, Meetings, Minutes, and Reports, makes sense to the people who maintain these pages; they appreciate the automation of linking and archiving it provides.

Files as tertiary content:
This is where a flyer, a report, or some other document is attached as supporting material on a page or a story. It also includes cross-referencing a form that is already uploaded as primary or secondary content, when writing a timely article or update for example. IMCE is used to upload and place images. Disknode is used to cross-reference files.

Shortcomings:
Some users really want to upload files to a directory and then attach or categorize them. They hate having to go through the Web interface when they have a lot of files to publish, especially a lot of large files. Enabling sFTP or WebDav can mitigate that problem, but there are benefits to having drupal aware of the files.

Note: Every method we used for uploading files was buggy in one way or another, and they all wanted to store the files in some different directory structure. The system in place still suffers from a filefield bug: http://drupal.org/node/166676. It's possible that there are much better options in drupal 5.7, and things like the core file upload might not suffer from whatever made us abandon it which I can't precisely remember right now. So, rather than explaining the past I'll switch to the future, but don't overlook obvious choices :)

Requirements:
The way I see it, the desired functionality looks like:

* get files uploaded more easily than uploading them one at a time via a Web form
* have file access that lets all authors and editors view and cross-reference each others' files but not update or replace them
* because of that cross-referencing, have a pathauto style naming process that allows automatic naming in meaningful ways
* provide a consistent experience for uploading/cross-referencing images, node attachments, and primary/secondary files
* assist in preventing multiple copies of the same document from being present on the site (e.g., noisily allow the user to overwrite a file of the same name or re-name their current file rather than silently appending a _0 or _1)
* access restrict some documents

Some features of disknode that were appealing (which the CCK types didn't have access to):

* The ability to update all the links to a file if you moved or renamed it (which never applied to links in the body of a node)
* A warning about all the nodes that reference a file which would break if you deleted, moved, or renamed it.

State of the Files Address

eliza411's picture

This is Angie Byron's response to the write up above, posted with her permission. There were at least three people at the meeting struggling with file management who might find this useful.


Unfortunately, file management is an area of Drupal that is (and has been historically) in need of a bit of TLC. And it shows, since by browsing through the list of file management modules @ http://drupal.org/project/Modules/category/62, there are several examples of people saying "Screw this. I can write a better file management API. Use $module instead!" Asset, WebFM, Disk Node, etc. are all examples of this.

The danger, of course, is that these "other" file management systems are:
a) wildly inconsistent in what they offer
b) wildly inconsistent in how they solve various problems
c) independent development "silos", separate from core, and dependent on the maintainers of those systems sticking around.

Drupal core's upload.module is the only one guaranteed to be updated through the ages, with CCK's file field a close second, since it's essentially "as core as you can be without being core." Venturing outside of these two paths is dangerous, but some kind of "module soup" (plus a healthy dose of new/custom development) will almost certainly be required to fulfill all of the requirements you have laid out here.

So what I'd strongly suggest first and foremost is prioritizing these requirements, and figuring out:

a) What is the bare minimum that needs to be rolled out with this upgrade in order to keep all of the various stakeholders from killing each other?

b) How much of the proposed functionality do they think they want vs. actually need? And is it possible, politically, to get them to back off on the optional stuff, in the interest of easier future maintenance of the site?

Now, with that out of the way... ;) Let's go through your requirements list of "file-related thingies."

Bulk file uploads:
- While there are various solutions around for bulk-uploading images, unfortunately none of these solutions look to be generalized for files. You can make a multi-value file upload field in CCK, but it's clunky, and in Drupal 5 you get 3 of those and that's it. :\
- There are "Allow multiple uploads" feature requests for just about every utility module imaginable. They're all active. :P

Automatic file renaming:
No existing solution for this that I could find. http://drupal.org/project/uploadpath could be something to build from, though. It uses the "Token" module, same as Pathauto, and lets you specify a pattern for what subdirectory a file should be uploaded to. For example:

[type]/[user-name]

Would upload forms to:

files/upload_file/admin/form-name.pdf
files/upload_file/bob/some-other-form.pdf

and so on.

Two issues:
1. It doesn't touch renaming the files themselves, only controlling which sub-directory they're uploaded to. This would actually make your duplicate form problem worse, because instead of them being _0 or _1, they'd now be files/forms/something/something/duplicate-name.pdf
2. It seems to be suffering from "overloaded-module-maintainer-itis" (Larry's a great guy, but he has his finger in one too many pies ;)).

Finely-grained file access:
Hoooo boy. ;)

So Drupal core offers an "all or nothing" private/public file solution. Your files are either public (in which case Apache serves all files just as it would normal .css files or whatever) or private (in which case Drupal serves them itself, which allows it to do permissions checking).

Naturally, lots of people thought that was dumb, and implemented their own workarounds. There's http://drupal.org/project/private_upload which lets you control a "Private" checkbox on a per-file basis, using core's upload.module. On the CCK side of things, CCK Field Permissions would let you set create/edit & delete/view access on those file fields that are part of the primary/secondary file types. This wouldn't affect the underlying file, however.

With private files, access is determined based on access to the node to which the file is attached, so when using a node_access module like Taxonomy Access Control, you have the ability to control who can view files to the same level of control that you have to control who can view the nodes.

Private files only control viewing of the files, however. If you can edit the node, you can edit the files attached to them as well. Further, Drupal's permission system is also role-focused, rather than user-focused, so implementing "Mary can view Susan's files but can't upload them" isn't really something that Drupal could deal real well without blowing up your already gigantic roles page even more. :\

This might be something you could handle with Unix system-level file permissions, if you had some smart scripts on the back-end. Nothing that I'm aware of in Drupal can handle this currently, however.

Consistent UI
*I'm still searching for some kind of 'magic bullet' here, but something that *might
be worth looking at, both from a bulk-file-upload standpoint as well as a "unified user experience" standpoint, is SWFUpload: http://demo.swfupload.org/ which provides a Flash/JS-powered local file browser that looks exactly like your OS file browser, but allows multiple file uploads via AJAX (you have to ctrl+click them though; not sure what kind of learning curve that would represent for your users). There's a module for it at http://drupal.org/project/swfupload, but it doesn't seem to work overly well, or that is to say "at all." :\ It's also specific to core upload module, although it might not be terribly hard to create a patch to make this work for CCK File fields and IMCE too.

*File link breakage notification
*That is indeed a cool feature, and I'm not aware of any other modules that offer it. :\ I do know that in my test environment, Disknode completely forgot about the file I pointed it to between saving and re-editing the node, so I'm wondering how critical this feature is. :P

Date change?

aaron's picture

Glad I looked at the page; I almost hit submit to purchase tickets for the 17th through the 21st. Could someone confirm to me the dates are correct now?

I'm coming into Portland International Airport. I'll get tickets to be there that week.

Thanks,
Aaron Winborn
AaronWinborn.com (my blog)
Advomatic (my work)
Drupal Multimedia (my book)

Aaron Winborn
Drupal Multimedia (my book, available now!)
AaronWinborn.com
Advomatic

since you're the only one i

drewish's picture

since you're the only one i know of that's coming to town specifically for this, i bumped the dates to jive with the dates you'll be here. i'll see you tomorrow.

Re: Drupal core's upload.module

pasada's picture

@Cristiano: Did you intend to post this as a quote of Angie Byron's comments above? (http://groups.drupal.org/node/11810#comment-39681)