Posted by djudd on September 18, 2009 at 2:37pm
Obviously, newspapers tend to upload a lot of images.
Our site is using imagefield and imagecache to upload images and to size them dynamically where needed. In our first month, we've already uploaded about 500 photos, all of which are just residing in sites/default/files and then the various imagecache subdirectories.
It dawned on me that the files folder is going to get ridiculously huge, upwards of thousands of photos in the first year alone.
I was curious how other newspapers are handling this.
Comments
Ok and how You use these photos...
Let's say other time someone upload a photo that already is on server...
Is there a possibility to check if a photo of some category I need is already loaded?
If You will have 1000 photos choosing them by watching may take a lot of time...
And uploading without checking is bad - You may upload a photo that already exist and You miss You data You already have...
konrad, that makes total
konrad, that makes total sense. However, in the four years I have worked for this paper, I don't think we've ever repurposed a photo.
This is a small town and our news tends to be fresh every day because we don't have a lot of continuing stories. So the odds of us needing a photo we already ran are pretty thin.
Imagefield
We're using imagefield as well, and considering photos to be properties of a (news story) CCK node, so the repurposing stuff that konrad is interested in doing is not possible. We're actually looking at using our (proprietary) Spotted photo gallery software as a photo library repository providing a Web service to Drupal, but I'm not sure how that's going to shake out just yet.
As for the original question: I know that in theory, retrieval of a file from a deep directory tree is faster than pulling the same item from a really big pile in a single directory, but in practice, how important is that? I used to worry about that a lot when I was stuffing data in to a DOS-formatted slow disk with an 8-megahertz CPU, but we're living in an era of 3-gigahertz CPUs, fast disks and Unix RAM-buffered filesystems that I don't even begin to understand.
I don't know how important it
I don't know how important it will be into the future.
I've seen a machine, even a fast one, grind to a screeching halt with 100,000+ files in one directory and that directory was accessed, so I'm just not sure what it might do to a web server once the number of files gets to that point.
One thing is for sure, it's something I'd like to think about before it happens, haha.
Token Module
We were worried about the same thing. We used the token module so all images are stored by year/month. Images are attached using a cck field.
SSP?
I have EXACTLY the same concerns as we prepare to move our paper onto Drupal. I am trying to get Slideshowpro.net working for us. My goal is to use their Director server product (sweet awesomeness in a can) to store and manage all our images. Most of our stories will have multiple images and we want to have them in the upper right in an expandable or pop-up slide show. This product does that wonderfully. Everything is handled by a single embedded swf which links up with the director server via xml for its content.
The workflow for writers would be:
Create a new Story in Drupal and click on a custom link in the content page that links to the Director frontend. Here they would upload all the images for the story (in bulk via browser). And name the album. Director will spit out an identifier to the album. They would copy the album ID info and close out of Director. Now back at the Drupal new story page they would paste the ID in and compose the rest of the story (text etc.). The theme would hand the XML feed for the proper director album into the embedded swf (also in the theme) and viola!
Has anyone else here done anything like this? I could use any ‘lessons learned’ stuff you might have for me since this is all just vapor ware for us at the moment. Also is there something I am missing as to why this approach would be bad?
Thanks
*_*
I don't know if its the best approach, but since articles, once uploaded to the site, will not change, right?
If they don't, they can be converted to pdfs, with images being links. By this way, no need to manage images separately any more.
Also, publishing an archive is more manageable.
BeautifulMind
akempler, could you explain
akempler, could you explain that a little more please? Your solution sounds perfect!
I took at look at the imagefield cck field, and I can't see any token for month and year that the system will allow. I'm on D6.13 at the moment.
I do see one for current system time, [site-date], but that drills a little deeper than I would like.
Store imagefield images in directory by month
This works for me. Edit the imagefield. In path settings put: photo/[site-date-yyyy]/[site-date-mm]
So photos for this month are stored in: photo/2009/09
I have them resized to 600px for the long side.
I use imagecache to generate thumbs of 80x80 px and node view versions of 465 px.
This creates directories like: imagecache/thumb_80x80/photo/2009/09
I keep the 600px versions just in case we want to use bigger images in the future. What do you consider a future proof archive size? I hesitate if 800 or 1000 might be better.
custom token
We created a custom token by putting the following code in one of our modules.
This is for Drupal 5.x but should work in Drupal 6.x
<?php
/**
* Add tokens for saving uploaded files to YYYY/MM/DD folders
*/
// function celsius_iptc_token_values($type, $object, $options) {
function celsius_iptc_token_values($type) {
if ($type == 'user') {
$user = $object;
$tokens['dir-date'] = "files/".date('Y/m/d');
return $tokens;
}
}
function celsius_iptc_token_list($type) {
if ($type == 'user' || $type == 'all') {
$tokens['global']['dir-date'] = t("Stores uploaded files in /yyyy/mm/dd/ folder");
return $tokens;
}
}
?>
I'm not sure exactly why, but
I'm not sure exactly why, but the use of [site-date-yyyy]/[site-date-mm] tokens fails for me and causes a form error when uploading a photo.
However, I have been able to use the more rudimentary [site-date] token, and while that breaks it down further than I would have liked by adding a time code as well, it certainly solves the problem of everything uploading to one folder.
Thanks for the insight on this one guys. I should have considered using tokens anywhere, but I'm still very wet behind the ears with Drupal and learning on my feet.
My two cents
Hey all, I just joined this group. Cool topic. I work with some magazines in Vancouver.
We use imagefiled as well and we have similar issues as noted above. Recently I've been migrating some of my other sites to IMCE + IMCE CCK Imagefield. I wrote a howto about it here: http://www.verbosity.ca/website-development/managing-all-your-images-drupal. It is a different approach but one I really like. Hope it helps someone out.
How Oracle is doing it
Hi,
I'm working in a publishing company, too. (12 issues, 120.000 prints).
Unfortunately I lost the new website project (drupal) against a service provider (their own development) - doesn't matter at all.
We have only one important system and this is our local publishing software (not web based) but there is a good concept inside:
Oracle stores all the pictures as blobs in it's tables.
Every night a workflow comes and swaps these images from the database to the file system and leaves a pointer in the database.
We've a Centera storage where roughly one terabytes of these pictures resides.
Maybe for your solution it's a bit oversized, but if you start talking about 100.000+ (we get around 3000 each day), maybe it would be a good idea to search for a linux based swap to storage / archive solution with leaving pointers (like links or inodes) on the current filesystem.
Hope the idea helps anyone.
For smaller projects I would do it with /year/month/picture.jpg
Often the authors know the year and sometimes the month to start a search.
Bye, Chris