Importing from a csv file

We encourage users to post events happening in the community to the community events group on https://www.drupal.org.
jsimonis's picture

I have a csv file that has all the data for our nodes. The node is primarily cck fields, plus the standard title and body. The fields are all one of the following:

text box
text area
content taxonomy (checkboxes)
a few yes/no radios
checkboxes
one hierarchial select

All the text boxes and text areas import with node_import without problem. Everything else gives us a unicode error. I've been trying to get assistance from the module's maintainer(s) without luck. I've got to get this imported. I've been fighting with this for weeks. By now I could have hand inputted them (and it's a large number), but he needs import up and running for future imports.

Any ideas? I've tried taking the csv file, opening it in other programs, and ensuring the new file is UTF8. I still get the unicode errors on everything that isn't a text field or text area.

Comments

Have you tried the Migrate module?

ensignavenger's picture

Have you tried using the migrate module? http://drupal.org/project/migrate

Feeds module

bonobo's picture

We've imported cvs files using Feeds - it worked pretty easily.

+1 for Feeds, but...

mudsurfer's picture

I'm a fan of feeds as well, and have used to import data from csv files to nodes on a couple of projects, but for the life of me, I cannot get it to import dates effectively into a cck date field. I get close, but even then it converts to UTC (GMT) on import, no mattervwhat I do...
Other fields no problem, Title, description, cck text field - all import really easily.
Anyone else successfully using Feeds with cck date field?

Have you talked about this in

R.J. Steinert's picture

Have you talked about this in the Feeds issue queue?

Self-Diagnosis

mudsurfer's picture

rjstatic: I had asked around on chat and trawled the groups postings first - I dont like posting in the issue queue until I'm sure it's an issue, and not just me doing something stupid. There does not seem to be anyone else around mentioning my problems, which leads me to believe it's just me :)
That being said - even with my work arounds in excel, the Feeds solution is the best I've found! Feeds is cool.

I just imported 26,000 nodes

R.J. Steinert's picture

I just imported 26,000 nodes from CSV using Feeds. +1 for feeds :)

Comparison table

Frank Ralf's picture

Migrate is a great module. See Comparison of Content and User Import and Export Modules for some further pointers.

If it's really a Unicode related problem you might find some useful information at http://docs.moodle.org/en/UTF-8_and_BOM

hth
Frank

Nice thing with feeds is that

kthull's picture

Nice thing with feeds is that once you do get it set up, assuming your future imports are of the same structure, your ground work is already covered. And (while I haven't tested this myself) you can tell it to update nodes that have been previously imported.

Mudsurfer: any chance your date issues is linked to the format of your date field vs the format of the csv date? I know there's a couple cck date formats to choose from and maybe that's where the problem lies. Just going out on a limb with that though.

I build websites, push pixels, move type, make media, plan camps, tap mana, roll for initiative, eat like a boss, chase food trucks and like an #eggoneverything. Sponsor my travel and I will record and post your camp sessions!

Update re Date fields with Feeds

mudsurfer's picture

kthull: Update - you are correct, it was date field format related (partly) - I am now using the "date" format rather than datestamp, and am having luck with the import. However the other big problem I discovered was that once I was abl eto import the date fields - the content I'm importing is getting converted to local site time - no matter what settings I use for time zone handling - at the moment, the way it is working for me - I have "No time zone handling" set on that field - but when I create the CSV data, I have a spreadsheet that removes 11hrs from the time I want to use, and places the adjusted time in the CSV file, Then when I upload via feeds, everything gets populated, and the date field is adjusted by Drupal and adds on the 11hrs again (my site time is currently UTC+11:00).
Basically - no matter what I tried, Drupal/feeds is expecting me to load the date data in GMT/UTC.
Complete pain in the buttinski - however this spreadsheet work around is working just now... and right now I just need it to work!
Hopefully I can revisit this in a couple of weeks and see why /where I can stop drupal from converting that data .
PS: Also I had issues with the saving the CSV file until I worked out that I need to save the file in UTF-8 format, I'm on windows7 and I use Text-Pad to do that, Excel is definately bad...

Import from CSV, seems like a simple thing...

wcndave's picture

I want to import some data to populate a particular content type, by creating new nodes, from Excel, with about 6 fields.

Seems like a really simple thing to want to do.

Well, after spending hours and hours googling and reading forums, i summarised as follows:

  • node import, not available for 7, doesn't even install
  • direct SQL inserts, well, it creates the node, and i put data in the field_data_field_XXX and field_revision_field_xxx however it never appears in the node, under view / edit etc...
  • migrate - looks like you have to write lots of code - seems a bit unnecessary, I want to 100 records 4 times a year, would be quicker to do data entry..
  • Feeds, thought I'd hit the silver bullet here, and as I set it up, i got more and more excited.... however it simply doesn't work (for me) 1. i cannot set up the mappings, a number of my fields are just not there. They seem to be all the list ones. 2. One field, a date field did import ok (I had set it to yyyy-mm-ddT23:00:00 manually in the CSV), however even the title did not import ok, it just populated the node with the type name...

can anyone, anywhere, please tell me how to get a simple CSV file, which can contain the SQL-ready data into nodes? I am happy to do SQL queries to insert, and this was my first approach, however does not work when i do it, and I cannot find any answers, having now spent 40% of the last two weeks searching and trying....

Edit: I also tried using Firefox developer bar to turn post to get, record the URL, and then use /node/add to "post" the data as a series of create URLs, however that did not work either, i think the form_id to prevent cross site request forgery prevents this...

Your mileage may vary, but I

coderintherye's picture

Your mileage may vary, but I always inevitably turn back to creating nodes manually via a custom module, example for doing so is here: http://drupal.org/node/889058
It's pretty easy once you get the hang of it. Just fopen or file_get_contents or however you want to bring your data in, then explode by comma, then stick loop through and insert your data.

Drupal evangelist.
www.CoderintheRye.com

What about data outside of the direct node

wcndave's picture

Eg extra cck fields that i have added? can these just be added to that script as $node->some_field = 'some data';
?

Almost, $node->some_field[0][

coderintherye's picture

Almost,
$node->some_field[0]['value'] = 'some data';

Usually. Some fields, such as images, may have more to add to in the array.

Drupal evangelist.
www.CoderintheRye.com

Almost, $node->some_field[0][

coderintherye's picture

Almost,
$node->some_field[0]['value'] = 'some data';

Usually. Some fields, such as images, may have more to add to in the array.

Drupal evangelist.
www.CoderintheRye.com

Almost almost ;-)

wcndave's picture

I had to put

$node->some_field['und'][0]['value'] = 'some data';