Beta testers wanted

Posted by Jaza on August 15, 2006 at 5:42pm

Re-posting from the development mailing list:

The Import / Export API has finally reached what I would consider to be 'beta quality'. The module currently supports import and export of all entities in Drupal core (e.g. users, nodes, comments, terms/vocabs), as well as CCK nodes / fields / node types. All supported entities can be imported or exported in XML or CSV format.

The Summer of Code officially finishes next Monday (21st Aug). I would appreciate it very much if anyone who has the time (and the curiosity!) could help out with testing my work. The module is compatible with Drupal 4.7 stable (although it has no 4.7 branch, only a HEAD branch), and it can be downloaded here:

http://drupal.org/project/importexportapi

How to install and test it:
1. Extract and copy the package into your site's '/modules' dir.
2. Enable both the 'importexportapi' and the 'importexportui' modules at 'admin/modules'.
3. Perform exports at 'admin/export'. Currently, exported data is simply outputted in a themed page.
4. Perform imports at 'admin/import'. Files for importing can be found (and played around with) in the '/testdata' dir of the package, and all valid files in this dir are listed for import in the UI. Needless to say, imports should be conducted in a test environment ONLY - there are bound to still be bugs, so don't do this on a production site.

Docs, notes, etc. can be found on the wiki:
http://basement.greenash.net.au/soc2006/

Discussion can be found on the group:
http://groups.drupal.org/soc-import-export-api

Please submit all bugs to the issue queue:
http://drupal.org/node/add/project_issue/importexportapi

Many thanks,
Jaza.

Comments

Wow!

Posted by dado on August 16, 2006 at 12:53pm

Initial impressions: Wow! On cursory examination I see
an extensive library of node types mapped
a robust import & export UI
import of dozens of things: filters, blocks, users, node TYPES, node access rules, any node type! Wow.
Import CSV or XML

There is so much here that it will likely take weeks if not months to get our head around it!

I think a quick "how-to" document would help for importing. I tried an import using a CSV file that just specified a few fields (title and a custom CCk text field 'description'), but nothing imported and no explanatory message. E.g. I cannot figure out what this means on the "Step 2: field mappings" page:
"Mytestnodetype (plural): *"

Also is there a way to disable import into taxonomy? This fieldset on the "Step 2: field mappings" page seems to require import of taxonomy terms
Raw import file for 'Taxonomy terms': *

Awesome work, Jaza!

First Beta Test

Posted by victorkane on August 16, 2006 at 4:06pm

Well, I am going to modestly assume the role of Beta Tester, and as such, will carry out the black box testing you propose in your 4 point "test case".

I will carry out 3 tests, one on an empty drupal installation on my development box, one on a populated prototype in a shared hosting environment, and one on another populated prototype on another shared hosting environment. In all cases with Drupal 4.7.3, installed manually (not with fantastico).

Results of first test, carried out with Apache 2 and MySql 4.1 on a Ubuntu development box.

Copied contents to modules directory.
Enabled both importexportapi and importexportui in admin/modules
Went to see if there was help available. Stubs were present.
Went to read the README and INSTALL files. None present.
Went to admin/access and gave access to test user.
Went to perform exports at admin/exports. Import and export items appeared on administer menu.
Followed following instructions for story, page and book nodes:
"Perform exports at 'admin/export'. Currently, exported data is simply outputted in a themed page."
Worked fine (I chose xml format for output with the default field mapping settings, and this was exported to a themed page as per specified test case observation point) for story, book (several chapters, 3 deep hierarchy, content (text only) included), user (accurate results), and for both book, page and story combined.
Imports, taken from testdata only, as per instructions.
Went to admin/import.
A little confusing here, what with the explanation given on the admin/import page on dependencies (unexplained) and the need to actually open the import file to see what entities were to be checked on the entities to import list, I was unsure how to proceed (fatal for a modest beta tester: you should give out a standard set of tests with instructions, so people can try them in different environments in an unambiguous form).
So, I opened the node_data.xml file and checked off the following entities:
vocabulary, term, comment, story, and I specified xml format.
On the next page I specified the hard coded "node_data.xml" option; and left the default (proposed) field mappings alone.
I got a zero here; results:

Your import has been completed. The results of the import can be seen below.

* user warning: You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near 'WHERE vid = 1' at line 1 query: SELECT name FROM WHERE vid = 1 in /var/www/drupal-4.7.3/includes/database.mysql.inc on line 120.
* user warning: You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near 'WHERE vid = 2' at line 1 query: SELECT name FROM WHERE vid = 2 in /var/www/drupal-4.7.3/includes/database.mysql.inc on line 120.
* user warning: Unknown column 'title' in 'field list' query: SELECT title FROM comments WHERE nid = 3 in /var/www/drupal-4.7.3/includes/database.mysql.inc on line 120.
* user warning: Unknown column 'title' in 'field list' query: SELECT title FROM comments WHERE nid = 3 in /var/www/drupal-4.7.3/includes/database.mysql.inc on line 120.
* user warning: Unknown column 'name' in 'field list' query: SELECT name FROM node_comment_statistics WHERE uid = 1 in /var/www/drupal-4.7.3/includes/database.mysql.inc on line 120.
* user warning: Unknown column 'mail' in 'field list' query: SELECT mail FROM node_comment_statistics WHERE uid = 1 in /var/www/drupal-4.7.3/includes/database.mysql.inc on line 120.
* user warning: Unknown column 'name' in 'field list' query: SELECT name FROM node_revisions WHERE uid = 1 in /var/www/drupal-4.7.3/includes/database.mysql.inc on line 120.
* user warning: Unknown column 'mail' in 'field list' query: SELECT mail FROM node_revisions WHERE uid = 1 in /var/www/drupal-4.7.3/includes/database.mysql.inc on line 120.
* user warning: Unknown column 'name' in 'field list' query: SELECT name FROM node_revisions WHERE format = 1 in /var/www/drupal-4.7.3/includes/database.mysql.inc on line 120.
* user warning: Unknown column 'title' in 'field list' query: SELECT title FROM term_node WHERE nid = 2 in /var/www/drupal-4.7.3/includes/database.mysql.inc on line 120.
* user warning: Unknown column 'name' in 'field list' query: SELECT name FROM term_node WHERE tid = 1 in /var/www/drupal-4.7.3/includes/database.mysql.inc on line 120.
* user warning: Unknown column 'title' in 'field list' query: SELECT title FROM term_node WHERE nid = 2 in /var/www/drupal-4.7.3/includes/database.mysql.inc on line 120.
* user warning: Unknown column 'name' in 'field list' query: SELECT name FROM term_node WHERE tid = 2 in /var/www/drupal-4.7.3/includes/database.mysql.inc on line 120.
* user warning: Unknown column 'name' in 'field list' query: SELECT name FROM node_comment_statistics WHERE uid = 1 in /var/www/drupal-4.7.3/includes/database.mysql.inc on line 120.
* user warning: Unknown column 'mail' in 'field list' query: SELECT mail FROM node_comment_statistics WHERE uid = 1 in /var/www/drupal-4.7.3/includes/database.mysql.inc on line 120.
* user warning: Unknown column 'name' in 'field list' query: SELECT name FROM node_revisions WHERE uid = 1 in /var/www/drupal-4.7.3/includes/database.mysql.inc on line 120.
* user warning: Unknown column 'mail' in 'field list' query: SELECT mail FROM node_revisions WHERE uid = 1 in /var/www/drupal-4.7.3/includes/database.mysql.inc on line 120.
* user warning: Unknown column 'name' in 'field list' query: SELECT name FROM node_revisions WHERE format = 1 in /var/www/drupal-4.7.3/includes/database.mysql.inc on line 120.
* user warning: Unknown column 'title' in 'field list' query: SELECT title FROM term_node WHERE nid = 3 in /var/www/drupal-4.7.3/includes/database.mysql.inc on line 120.
* user warning: Unknown column 'name' in 'field list' query: SELECT name FROM term_node WHERE tid = 3 in /var/www/drupal-4.7.3/includes/database.mysql.inc on line 120.
* user warning: Unknown column 'title' in 'field list' query: SELECT title FROM term_node WHERE nid = 3 in /var/www/drupal-4.7.3/includes/database.mysql.inc on line 120.
* user warning: Unknown column 'name' in 'field list' query: SELECT name FROM term_node WHERE tid = 4 in /var/www/drupal-4.7.3/includes/database.mysql.inc on line 120.

Let me know what to change, or if I made a mistake.

Victor Kane
http://awebfactory.com

By the way, the import was successful

Posted by victorkane on August 16, 2006 at 4:12pm

Since you did not specify observation points, I provided my own:

Go to administer >> content and see if you imported anything.
Yes!

Story: "Love" with 3 comments, with users mapped to local user id
Page: "Being a frog" with no comments, user mapped to local user id
Story: First post

Date and time were as per data file.

Victor Kane
http://awebfactory.com

Thanks for feedback

Posted by Jaza on August 17, 2006 at 3:55pm

Thanks very much for the testing that you did, Victor. You obviously put a fair bit of time and thought into your tests, and this is much appreciated.

Another user has experienced very similar problems/errors to you, and has filed a bug report here:

http://drupal.org/node/79091

As you can see from reading the thread, I have committed a number of changes to the module, which fix all of the errors that I was able to reproduce in my latest round of testing. As you can also see, I am not confident that I reproduced all the errors - there are very likely still some bugs lurking around, particularly in the 'db' 'put' engine (the part of the API that actually imports the data into the database). But hopefully, you should no longer be seeing the multitude of errors that you reported seeing previously.

Thanks also for your usability feedback. I am aware that the UI is not very user-friendly at the moment (even for tech-savvy beta testers such as yourself!) - unfortunately, I haven't been able to devote as much time to the UI as I have to the API. The API has been maturing and developing for about 3 months, whereas the UI is less than a week old ;-). But given time, I do hope to improve it.

FYI: currently, the import system simply displays the number '0' to indicate a successful import (not that it's really able to identify errors in import ATM - it basically always returns 0). I realise that this is neither informative nor user-friendly, and I hope to provide some more meaningful feedback on import (e.g. 'x no of stories and y number of terms imported successfully') when I get a chance. For now, you'll just have to live with the '0', and check the import results manually.

Jeremy Epstein - GreenAsh

csv export does not work so

Posted by Anonymous on August 21, 2006 at 3:06am

csv export does not work so well: 2 issues:
1) inability to export non trivial amounts of data.
2) inconsistent data fromat

1) I selected all available check boxes and then tried export: I got standard IE The page cannot be displayed
The page you are looking for is currently unavailable. The Web site might be experiencing technical difficulties, or you may need to adjust your browser settings.

2) asking for only book pages in csv format suceded, but the output format seemed to be inconsistent in the first few "lines" of the output. I verified the output by selecting the webpage contents, pasting into notepad, saving to a file and then opening with excel. Rows 7 to 163 in the resulting spreadhsheet seemed to make sense (first set of columns listed all revisions of the book pages, cols N to AH seemd to have a list of the book pages). But the first 7 rows of the output seemed to be messed up and did not follow the pattern of rows 8 to 163. The following is the raw csv text for the first 8 rows:

"","","","","","","","","","","","","","","","","","","","",""
"1","1","book","some things (modules) to download?","1","1","1154754193","1155442963","1","0","0","2","1","1154754193","","1","0","1","26","8","1156125552","2","2","1","also look for themes","Installing new themes
http://drupal.org/node/456

http://drupal.org/project/themes

Customization and theming
http://drupal.org/node/257","Installing new themes
http://drupal.org/node/456

http://drupal.org/project/themes

Customization and theming
http://drupal.org/node/257","","1155088167","1","2","2","1","0"
"1","1","1","some things (modules) to download?","Info on installing new modules:
Take note of putting them in sites/default/modules/ and not modules/
Click administer » access control. Scroll down to see if the module appears in the list and, if it does, give the appropriate permissions to roles.
Click administer » settings and see if the name of the module you just installed is in the list. If it is, click the module name and configure as appropriate.

http://drupal.org/node/70151

http://drupal.org/project/modules

http://drupal.org/handbook/modules","Info on installing new modules:
Take note of putting them in sites/default/modules/ and not modules/
Click administer » access control. Scroll down to see if the module appears in the list and, if it does, give the appropriate permissions to roles.
Click administer » settings and see if the name of the module you just installed is in the list. If it is, click the module name and configure as appropriate.

http://drupal.org/node/70151

http://drupal.org/project/modules

http://drupal.org/handbook/modules","","1155442963","1","1","1","4","0"
"2","2","book","also look for themes","1","1","1154755730","1155088167","1","0","0","2","2","1155336270","","4","1","2","14","0","1155336827","3","3","1","Gallery (G2) Info","Webpage Link:

http://gallery.menalto.com/","Webpage Link:

http://gallery.menalto.com/","","1154787871","1","3","3","0","0"
"3","3","book","Gallery (G2) Info","1","1","1154787871","1154787871","1","0","0","2","3","1154787871","","1","0","3","3","0","1154788030","4","4","1","Drupal Todos","Choices in Navigation - An Overview
http://drupal.org/node/55610
I defined a primary link to gallery page & it then placed a ""gallery"" menu top right (good!), as well as creating a new Primary Links menu on top on the previous menu on left. Not good!
Block: controlling content in the sidebars
http://drupal.org/handbook/modules/block
Menu: customize site navigation
http://drupal.org/handbook/modules/menu

figure out why there are 4 small randoms & 1 large random on left when in drupal nodes & then 4 small randoms, 1 large random under gallery navigation & an additional large random under gallery when i go to gallery
The second random was caused by having both the Gallry & the gallery navigation blocks enabled in drupal. the 4 small randoms was caused by having the gallery grid block enabled.
So you only really want to have 1 of these 3 enabled. which one is best?

find out which gallery block I want enabled.

find out about ""clean URIs""

Find out how to make urls embedded in pages to be hyperlinks: DONE! URLFlter module

set up email stuff - see php.ini

http://drupal.org/node/43794

set default front page
http://drupal.org/node/15364

","Choices in Navigation - An Overview
http://drupal.org/node/55610
I defined a primary link to gallery page & it then placed a ""gallery"" menu top right (good!), as well as creating a new Primary Links menu on top on the previous menu on left. Not good!
Block: controlling content in the sidebars
http://drupal.org/handbook/modules/block
Menu: customize site navigation
http://drupal.org/handbook/modules/menu

figure out why there are 4 small randoms & 1 large random on left when in drupal nodes & then 4 small randoms, 1 large random under gallery navigation & an additional large random under gallery when i go to gallery
","","1155264881","1","4","4","0","0"
"4","93","book","Drupal Todos","1","1","1154787943","1155952989","1","0","0","4","8","1","Drupal Todos","PHP page snippets
http://drupal.org/node/23220

Choices in Navigation - An Overview
http://drupal.org/node/55610
I defined a primary link to gallery page & it then placed a ""gallery"" menu top right (good!), as well as creating a new Primary Links menu on top on the previous menu on left. Not good! Got rid of it via administer blocks & disabled primary links.
Block: controlling content in the sidebars
http://drupal.org/handbook/modules/block
Menu: customize site navigation
http://drupal.org/handbook/modules/menu

added front page module
http://drupal.org/project/front
disabled front_page since it does not provide a full solution:
a) still needed to enable content so content menu entry showed up for visitors
b) embedded a link & it was not processed properly.

find out which gallery block I want enabled.

find out about ""clean URIs""

Find out how to make urls embedded in pages to be hyperlinks: DONE! URLFlter module

set up email stuff - see php.ini

http://drupal.org/node/43794

set default front page
http://drupal.org/node/15364

","PHP page snippets
http://drupal.org/node/23220

added front page module
http://drupal.org/project/front
disabled front_page since it does not provide a full solution:
","","1155272029","1","8","4","0","0","2","4","1155528300","","1","1","4","152","2","1156119937"
"4","22","4","Drupal Todos","PHP page snippets
http://drupal.org/node/23220

find out which gallery block I want enabled.

find out about ""clean URIs""
http://drupal.org/node/15365

Find out how to make urls embedded in pages to be hyperlinks: DONE! URLFlter module

set up email stuff - see php.ini

http://drupal.org/node/43794

set default front page
http://drupal.org/node/15364

","PHP page snippets
http://drupal.org/node/23220

added front page module
http://drupal.org/project/front
disabled front_page since it does not provide a full solution:
","","1155339246","1","22","4","0","0","5","5","book","Gallery2 & Drupal","1","1","1154788310","1155082432","1","0","0","2","5","1154788310","","1","0","5","12","0","1155358626"
"4","23","4","Drupal Todos","PHP page snippets
http://drupal.org/node/23220

find out which gallery block I want enabled.

find out about ""clean URIs""
http://drupal.org/node/15365

Find out how to make urls embedded in pages to be hyperlinks: DONE! URLFlter module

set up email stuff - see php.ini

http://drupal.org/node/43794

set default front page
http://drupal.org/node/15364

local links

","PHP page snippets
http://drupal.org/node/23220

added front page module
http://drupal.org/project/front
disabled front_page since it does not provide a full solution:
","","1155339285","1","23","4","0","0","6","84","book","Gallery todos","1","1","1154793553","1155442725","1","0","0","2","6","1154793553","","1","0","6","25","0","1155527570"

adding comments to the mix

Posted by Anonymous on August 21, 2006 at 3:29am

adding comments to the mix does not change things much.

Also want to add a slight clarification to the above: the first 6 rows are messed up, rows 7 to 163 are fine.

The comments appear to be added as yet another set of columns, with the first 6 rows being messed up. A pontential hint to the cause could be the first row being 21 "" fields. A "normal" row seems to have 34 columns.

First Tests

Posted by JacobSingh on August 24, 2006 at 11:35pm

Hi Jaza,

First, thanks for this great effort. We're trying to use for a real project with pretty complex requirements, and we'll be designing our own definition, so probably I'll be posting a lot this month :)

There was a bug with cck which I issued and patched.

Tests:

all tests with XML

CCK

I tried to do a cck import / export for just the type and field defs. No errors, but nothing happened either.

Blocks:

The export went fine, except it doesn't pick up user created blocks it seems.
Import seemed to do nothing again.

User Roles

I created a new role, gave it one permission "Administer blocks"
Ran export, looked good.

Ran the import, got the following:

Fatal error: Cannot unset string offsets in ...importexportapi_xml_get.inc on line 61

I'd like to work on resolving these issues, but I can't seem to get any import to do anything, so it's hard to start looking for the problem. Do you have any documented tests?

Thanks,
Jacob

Some fixed committed

Posted by Jaza on August 25, 2006 at 6:03am

There was a bug with cck which I issued and patched.

Thanks for that, Jacob. I have committed your patch.

Blocks: The export went fine, except it doesn't pick up user created blocks it seems.

User-created blocks were previously getting picked up, but were not getting their extra info (e.g. title, body text) outputted during export. This has now been fixed - custom blocks now have a separate definition to other blocks, which includes these additional fields.

Please file the other bugs that you've reported as issues, and I'll look into them when I can. Thanks for your efforts in testing this module - much appreciated. :-)

Jeremy Epstein - GreenAsh

another error

Posted by kevinwalsh on August 26, 2006 at 7:00am

Hi Jeremy,

I just installed the new (8/25/06) version and got the following:

Fatal error: Call to undefined function: content_types() in .../public_html/modules/importexportapi/definitions/contrib/cck/importexportapi_content.inc on line 114

Thanks for your work on this very important module!! Looking forward,

Kevin

/user/666
unlockedmedia.net

i think content_types() is a CCK function

Posted by dado on August 26, 2006 at 6:07pm

do you have CCK enabled? Is CCK required for this importexportapi?

cck is enabled and in

Posted by kevinwalsh on August 27, 2006 at 4:37pm

cck is enabled and in use.

thanks!

/user/666
unlockedmedia.net

Can't help you

Posted by Jaza on August 29, 2006 at 2:25am

I just checked, and the 4.7 and HEAD versions of CCK still both define the content_types() function. So if you have CCK installed, this function should definitely be available. And by the way, you should be using the 4.7 version of CCK (as well as Drupal 4.7) with the importexportapi (which only has a HEAD version - but the HEAD version is built for 4.7).

Jeremy Epstein - GreenAsh

I updated from an old

Posted by kevinwalsh on August 30, 2006 at 4:19pm

I updated from an old version of cck (though still 4.7) to a new version (29/08/06) and this seems to have resolved the error. Thanks!

/user/666
unlockedmedia.net

Import of Content Fields now working, can't seem to move nodes

Posted by Anonymous on August 30, 2006 at 12:48am

Hi Jaza,

Can you please explain how to move cck content from one install to another? This is what I gather, you have to first export / import the content type and field deffinitions. this seems to be working now.

After I've done this, the content type shows up in my data type selection list when importing, however, the export I made from the original install of Drupal doesn't get imported and the import feedback is a 0. When I look at the DB query log, I see a lot update queries which don't actually hit anything, but no inster queries...

What am I missing? I realize this is beta, and you have done some really amazing work, so don't take it the wrong way, but I think this project would be a lot easier to work on if the procedures returned some sort of feedback (other than null or 0). Since the nature of the project is such that nothing is hardcoded, variable names are variable and recursion is everywhere, without any type of log, error messages or error handling, someone who hasn't written the code would be hard pressed to know where to start. A good start would probably be to just log a stack trace (this function called with these params, etc)... I'd like to dig into the internals and figure it out for myself, but it's a bit intimidating

Thanks,
Jacob

Test Info - Ver 5.0

Posted by coupet on April 4, 2007 at 7:01pm

Export Story Entity (XML)

select Story entity and XML format
accept default values
Suggestions for User Interface
- Provide Header for node table (fields)
- Provide Header for node_comment_statistics table (fields)
- Header (collapsible) for Revisions (fields) & Taxonomy terms (fields) is perfect
Output one (1) XML file
- Is file story-collection.xml created in ../testdata directory ?

Export Story Entity (CSV)

select Story entity and CSV format
accept default values
Suggestions for User Interface
- Provide Header for node table (fields)
- Provide Header for node_comment_statistics table (fields)
- Header (collapsible) for Revisions (fields) & Taxonomy terms (fields) is perfect
Output three (3) CSV files
- story-collection.csv
- story-revisions.csv
- taxonomy-terms.csv
- Are files created in ../testdata directory ?

What are the files import requirements from csv?
- Are three (3) csv files required to import data that includes taxonomy information?
- Are node ID and term ID required to import data that includes taxonomy information?

Is it possible to import from one (1) csv file that include data as follows:
- Node without node ID - node ID should be created automatically if new node
- Node without term ID - taxonomy ID should be determined based on term name in one column in csv file

Very elegant Design and Concept.

Thanks, Darly

Hi, is this module able to

Posted by jurriaanroelofs on January 6, 2008 at 12:03am

Hi, is this module able to import nodes with imagefields?

peach from All Drupal Themes!

Running SooperThemes Premium Drupal Themes

No, it doesn't support CCK

Posted by christefano on January 6, 2008 at 1:06am

No, it doesn't support CCK fields unfortunately. See http://drupal.org/node/84148

Comments

Wow!

First Beta Test

By the way, the import was successful

Thanks for feedback

csv export does not work so

adding comments to the mix

First Tests

Tests:

CCK

Blocks:

User Roles

Some fixed committed

another error

i think content_types() is a CCK function

cck is enabled and in

Can't help you

I updated from an old

Import of Content Fields now working, can't seem to move nodes

Test Info - Ver 5.0

Export Story Entity (XML)

Export Story Entity (CSV)

Hi, is this module able to

No, it doesn't support CCK

Content migration, import, and export

Group organizers

New groups

Group notifications

Hot content this week