Content and User Import and Export Modules

It looks like the wiki is mostly complete and contains a lot of good information comparing a solid cross section of Import modules. Each of us have very valid approaches to handling imports and I don't want to detract from anyones efforts ... but there is an extremely valid argument that the Drupal community as a whole benefits from collaboration. I've been brainstorming with a few others about this issue and would like to reach out to you for your thoughts, opinions and interest in collaborating.

If you reflect interest we would like to provide an opportunity for us to get together and talk. Mike has a great group going at http://groups.drupal.org/migration-drupal that I think we could leverage and comments have been enabled on the wiki page at http://groups.drupal.org/node/21338 ... in addition to that I would like to propose a dual layered code sprint that happens the weekend of June 27th and 28th. We are having a Drupal Camp in Colorado with a capacity of up to 400 attendees. The most desireable outcome would be if each of you could make it to that event and we can come up with a way to make Drupal Imports rockk for everyone. The likelyhood of that happening in person may be a bit lofty so I would like to propose coupling the code sprint with a virtual code sprint similar to http://groups.drupal.org/node/18443

Following this approach would give us almost 2 months to first hash out amongst ourselves the direction we would like to take and organize community support.

I'd like to recieve your thoughts on this proposal before taking it further. Thanks so far for all the great work!

Well first, I'm not going to be in person anywhere (unless I get sponsorship or something like I did last code sprint). :-/

However, where I'm at is:
- enough players have been sniffing around to push import_html forward, so the d6 will be out and about within a month.
Certain API bits are moving forward, and certain functions have dropped out to become modular (contrib or glue modules can catch the raw data and the newly formed node pre-save and add their own data extraction additions via hooks)

I've watched the birth and death of import/export API (last activity April 2007) and am wary of looking for the one true answer. (I DID think it was possible in the beginning)
Migrating from different datasources - which is pretty much what the tools in this realm do - are unique jobs.

I've looked hard at my own and others code and I've not seen the point at which these jobs hit common ground. Node import is a different tool, and I use it myself, learned from it, and even patched it a bit. It's the closest to import_html of the lot - but I've not yet found a function that could be placed in common.

Maybe others can see some points of commonality. I'd be interested to hear. My docs are verbose on methodology and a walkthrough illustrates what can be done.

FYI, here's a slideshow of someones import experiments.
http://www.slideshare.net/emmajane/moving-in-how-to-port-your-content-fr...

.dan.

We've had a good amount of email communication between the authors of the WP2Drupal, Import/Export API, Migrate, and Import API modules. It sounds like the desire to start unifying our efforts is there ... now we just need to figure out what we will be doing and how. As expected, the idea of a code sprint where we can all collaborate is unlikely but we can still move forward. I understand that we all know our own modules quite well, but in an effort to fully understand the situation we need to take the time to evaluate the other modules in this category. The Migrate modules maintainer came up with these questions and I think they are a good starting point ... I'll be posting my answers after testing the modules listed above:

"As a starting point, I think an interesting exercise would be for each of us to look over the various modules in the same category, then write up:

1. What do I consider the single most important unique value of my module?
2. For each of the other modules, what's the one aspect I would most want to add to my own module?

If we can first converge on the unique value of each of these modules, we can move from there to figuring out how to make them work together..."

Sorry I've been quiet here - I was off on vacation last week, and I'm slowly getting caught up. I haven't had time yet to look closely at the other modules. But, some quick thoughts on the prospects of a common framework...

My experience doing large-scale migrations - not just with Drupal, but also with a big ERP (payroll/HR) migration a couple of years ago - is that the full process needs to look something like:

1. Analysis - The incoming data needs to be fully understood. Even in the rare event that it's documented, the real data never exactly matches the documentation.
2. Consolidation - Bring the incoming data into a common form.
3. Import - The actual creation of the result objects from the common form. This is not a one-time event, this is a process to be developed, so you need to be able to easily back out and redo the import process as you refine it (fix bugs, add features, etc.).
4. Auditing - how do you validate the results?

Obviously, not all stages need be implemented in a single module. But, by separating input (Analysis and Consolidation) from output (Import, Auditing), I do think one can have a common framework, dealing with various sources (HTML, Wordpress, etc.) independently (more-or-less) from destinations (nodes, users, etc.).

Mike Ryan
http://mikeryan.name/

I agree with Mikes analysis of the phases - it's certainly the way I go about it.
Steps 1-2 is the hard part - and is the bit that's different for everybody.
There may be some crossover we can do between the approaches for step 3 however. A queue of pending items, preview and rollback.

FWIW, my 'common form' for import_html is vanilla XHTML with a bunch of semantic meta tags and a few microformats. Once I've got from 'clunky data' to standardized clean data, I know the import will go smooth.
There is a second 'common form' inside step 3 - the Drupal PHP object itself - which is what some import mechanisms go straight to. I find that a bit too close to the bone and (although I do it) putting too much care into adding elements to $node always feels like I'm bypassing the API and almost touching the DB directly.
Maybe we just need better CRUD tools.

With taxonomy_xml, (import/export taxonomy terms) I broke the different formats up into their own libraries - which scan the input and return a queue of almost-complete Drupal $term objects. For that one, queuing and partial imports was harder as I needed internal references most of the time. I needed to save a term to get its ID so that another term could point to it as parent etc.

Just thought I'd say hello :)

I've been looking for an area to get involved with Drupal for a while, and seeing as I wrote the ImpEx for vBulletin (and 120 migration systems) this might be a good place to start !

Just reading over what is going on and getting up to speed, so I can lend a hand with some thoughts.

Cheers,
Jerry

http://drupal.org/project/migrator (in development)

http://drupal.org/project/transformations
http://drupal.org/project/transformations_drupal
http://drupal.org/project/transformations_csv
http://drupal.org/project/fieldtool

http://drupal.org/project/deploy

Export:
http://drupal.org/project/export_docbook

Daniel F. Kudwien
unleashed mind

Righty looks like a combination of transformations sitting on top of the Import / Export API.

See also: http://drupal.org/project/middleware

Hi all, been using Drupal since 4.6, used to make some small contributions to d.o here and there quite some time ago (satori at abc3400 dot be).

Anyways, I've done a major edit of the modules tables. Apparently, there's no way to view the revisions nor the logs, so I'm reposting my log message here for posterity:

Massive overhaul;
Grouped and ordered table rows logically, added links where applicable (eg. module names), added lots of info regarding modules' releases and dev status, reordered field text for global consistency, etc...
Still loads to do though, eg. cleaning up the "Files Imports " and "Internet Imports" rows. (Get rid of 'em? Could all be merged with the "Input" row...)

MODULE MAINTAINERS: I've taken the utmost care not to let any data get lost or corrupted, but please review, just in case.

Peace

I've been evaluating all of this information for a long time and have come to the conclusion that it would be in the best interest of the Drupal community to keep efforts focused on the best solution. For that reason I've discontinued the Import module and placed a reference on it's project page to the Migrate module. I still don't think the Migrate module is where it needs to be but it is leaps and bounds above the other modules in regards to planning and approach. I'll be using Migrate with all of my upcoming projects and look forward to submitting patches.

Hi, I'm currently developing a Drupal to Drupal synchronization/migration module, should it stands in this list?
See it there http://drupal.org/project/yamm

Pierre.

My brain literally just popped from the weight of my jaw hanging open in disbelief.

@pounard - Please add it - it looks very promising!

@cyberswat - Thanks for that note. I'm sorry that I never did do a really thorough review of all the modules covered here, although I did take a quick look at each of them. I'm not convinced there is a "single solution" - there is room for different domains (Drupal-to-Drupal migration/synchronization vs. import from external CMSs, not to mention the export side), and different approaches (quick-and-simple vs. kitchen-sink). I do feel Migrate is the best available external-CMS/kitchen-sink solution (naturally, since I've built it to meet my needs), but there's room for other modules.

I think it is still very valuable to maintain comparisons among modules in this general space, both to help people find the best module for their particular situation and to help developers avoid reinventing the wheel. Let's keep it up - but let's talk about the scalability of the chart, which is difficult to manage - can we do a Google spreadsheet or something like that?

Kevin, I'm looking forward to seeing your improvements to Migrate.

Thanks.

Mike Ryan
http://mikeryan.name/

@mikeryan did it.

@all My module is more like a Drupal to Drupal mass synchronization module than really an import/export module, I think some topics on general table are not really revelant for this module.

EDIT: typo

Pierre.

On small screens, blocks cover the right side of the matrix. This is quite annoying to fully read it I had to use firebug to put the whole blocks part as "display: none;", not much elegant.

Pierre.

http://drupal.org/project/node_export It's not in the table now, and I just used it yesterday and found it pretty helpful for Drupal-to-Drupal data migration. I'll try to add it in, although I'm not associated with the project at all. (It turned out to be the best solution for moving a webform from one site to another.)

This is a big chart and I'm not seeing one of the main deciding factors -- can import profiles be saved?

Eg. user import and backup and migrate - both permit saved profiles, which allows a site dev to preconfigure some and then pass the routine along to a site admin and it covers a wide range of abilities of that site-admin. I can't pass along 'node import' with the same ease at all.

This variable is perhaps more meaningful than 'target audience'

How about 'saved import profiles'?

The recent case study on the front page of drupal.org, Trócaire: Working for a Just World, says:

"we used Migrate to pull in the data from the existing CMS"

(...)

"The decision was taken to switch to Drupal, but there was still the issue of migrating all the existing content, files and users to Drupal. Using the excellent Migrate and Table Wizard modules, created by Mike Ryan and Moshe Weitzman of Cyrve, this task was made much easier. Over 2000 pages, 500+ files, almost 600 taxonomy terms and close to 100 users were successfully migrated across."

(...)

"Migrate, Table Wizard and Schema - allowed relatively pain-free migration of all existing content and users to Drupal."

(...)

"Trócaire were happy to contribute back all custom code and patches developed during the project to Drupal. Here are some of the main ones:"

(...)

"Migrate - #484404, #551254, #541700, #540908, #411156, #525996, #426824 and #520264."