Posted by jonraedeke on September 17, 2012 at 3:01pm
Government Drupal Folks,
Does anyone have experience grabbing compressed csv or text files on cron run and mapping to fields using the Feeds module or any other method?
Using Drupal 7.15 and Feeds 7.x-2.x-dev (Sept 8), I am able to successfully fetch a .txt file from a url and map it to my fields. The data source I need is at http://www.msha.gov/OpenGovernmentData/DataSets/Mines.zip is obviously a compressed file. When I attempt to fetch that, the Feeds module seems to try to unpack the contents and creates the correct number of nodes, but the fields are all blank. I can't figure out if Feeds supports compressed files. All I see is an issue for D6 http://drupal.org/node/682102. I tried to implement that code to no avail.
Any suggestions?
Comments
My first suggestion would be
My first suggestion would be to request MSHA to provide an uncompressed version on their website, how hard could that be. ;-)
Second suggestion would be to setup a scheduled process (cron) to grab the .zip file, extract it and then have feeds import from that source. You could probably even use a Drush command to do the import (assuming Feeds comes with Drush integration) from the same scheduled process.
I don't see any evidence in the git repository for Feeds that it includes code for handling zip or other compressed files.
Good luck!
I might wait on this
Tim - Thanks for the recommendations!
It turns out that the data is a bit of an overkill for what I'm trying to do. There are about 80,000 records and I'm only using them to speed up content migration for less than 3,000 entries. I think I will only import it once as a csv and then delete the remaining unused records with views bulk operations. I'll hold off on trying to import all these automatically since it will be a considerable drag on the site.
Those of us using feeds from
Those of us using feeds from merchants often find they're compressed as zip or gzip, so finding a relatively easy and pain-free solution to regularly importing feeds and updating nodes would be great.
I'm not a developer, so I'm not entirely sure of how to script such a solution. If Feeds can't handle compressed feeds, are there any off-the-shelf scripts out there that can help?