There are a multitude of different import and export modules in existence to handle the complex task of moving data between other systems and Drupal. This wiki is an attempt to focus the efforts of the module developers and provide a handy guide for users looking to make a decision.
Importing/Exporting/Transforming data is a complex process and each of these modules approach it in a different way.
Comparison of import/export modules:
- Import and Export (Generic)
- Import and Export (Drupal to Drupal)
- Import (Generic)
- Import (Source-Specific)
- Export
- See also
- Other modules (not yet reviewed)
Import and Export (Generic)
Collaboration |
|||
---|---|---|---|
Module | Import / Export API | Taxonomy Import/Export | Transformations |
Project page links here | |||
Communicated with other maintainer(s) | / | ? | / |
General Info |
|||
Module | Import / Export API | Taxonomy Import/Export | Transformations |
Maintained | Limited | Yes | Kinda |
Releases (Status) | d6 (dev), d5 (stable), d4.7 (stable) | d6 (stable), d5 (stable) | d6 (beta) |
Dependencies | Drupal Core | ? | CTools (only for UI), jQuery UI recommended |
Target Audience | Developer | ? | Site Admin, Developer |
Documentation | API (extensive) | Yes | API (extensive), End User (rudimentary) |
Weight/Drupal Core Upgrade Path | Heavy, but easy to upgrade | ? | Complex but modular |
Track Record | Proven against several thousand records (not highly scalable) | ? | Proven against import of 200 records (CSV file), converted to Drupal nodes. (erm.) |
Functionality |
|||
Module | Import / Export API | Taxonomy Import/Export | Transformations |
Interface | API, UI | ? | API, UI |
Input | CSV, XML, or another Drupal database (can be extended to support other formats) | XML, CSV, TCS, RDF | CSV, XML (via XPath), can be extended to support other formats. |
Internet Imports | Doable through hooks | ? | Not yet! |
Interoperability | Supports CCK. Can support any module through hooks. | ? | Plugin framework for providing whatever functionality. Current extension modules support CSV, XML, Drupal nodes. |
Processing (Batch, Cron, Single) | Single (can be scripted to run via cron or in batch) | ? | Single |
Output | Anything in the Drupal database, XML, CSV... (can be extended to other formats) | XML, RDF | Nodes (using Transformations -- Drupal data), ... (or any type that can be processed by plugins) |
Reporting | Per-import pass/fail only (not stored) | ? | Per-import pass/fail only (not stored) |
Settings saved for reuse | ? | ? | ? |
Import and Export (Drupal to Drupal)
Collaboration |
|||||
---|---|---|---|---|---|
Module | Backup and Migrate | Data Export Import | Deploy | Node Export | Yamm |
Project page links here | No | No | No | No | Yes |
Communicated with other maintainer(s) | ? | ? | ? | ? | / |
General Info |
|||||
Module | Backup and Migrate | Data Export Import | Deploy | Node Export | Yamm |
Maintained | Yes | Yes | Yes | Yes | Yes |
Releases (Status) | d7 (stable), d6 (stable), d5 (stable) | d6 (stable), d7 (stable) | d6 (dev) | d7 (stable), d6 (stable) | d6 (beta1) |
Dependencies | none | none | ? | UUID | DataSync, PHP 5.2 |
Target Audience | Site Admin | Site Admin, Developer | ? | ? | Site Admin, Developer |
Documentation | README.txt | README.txt | Handbook pages | ? | API (extensive), End User (enough) |
Weight/Drupal Core Upgrade Path | ? | ? | ? | ? | Not that complex |
Track Record | ? | ? | ? | ? | Works for maintainers, need comunity testing |
Functionality |
|||||
Module | Backup and Migrate | Data Export Import | Deploy | Node Export | Yamm |
Interface | UI | UI | ? | ? | API, UI |
Input | Database dump (SQL file) | Will export Users, Taxonomy terms and Nodes to dataset files. When these dataset files are imported the entities imported will have the same ID's as when they were exported. NB - This module will delete excess entities which are in the receiving Drupal site. The idea is that the receiving the receiving instance will end up with an exact copy of the data. (Think rsync with the --delete option set). | Drupal content and configuration | Drupal nodes | Any Drupal data on site, content type, nodes, users, vocabulary in core, easily extensible |
Internet Imports | ? | No | ? | ? | No |
Export Destinations | Server Directory, MySQL Database, FTP, Amazon S3 Bucket, E-Mail | Exports to and imports from dataset files. | ? | ? | ? |
Interoperability | ? | ? | ? | ? | Should support any module that extends node and users, API is highly and easily extensible |
Processing (Batch, Cron, Single) | Multiple backup schedules | Enabled for Drush. | ? | ? | Cron and Batch on system side (needs you to have a UNIX host) |
Output | Database dump (SQL file) | Dataset files. | Drupal content and configuration | Drupal nodes | Any supported Drupal data that server and client handles |
Reporting | ? | Feedback via the UI. | ? | ? | Weak watchdog output, DataSync reports |
Settings saved for reuse | Yes | N/A | ? | ? | ? |
Import (Generic)
Collaboration |
||||||||
---|---|---|---|---|---|---|---|---|
Module | CSV Parser | Feeds | Import | Import HTML | Migrate | Node Import | Taxonomy CSV Import | User Import |
Project page links here | No | No | Yes | Yes | Yes | Yes | No | Yes |
Communicated with other maintainer(s) | ? | ? | Migrate | Import / Export API | ? | / | ? | / |
General Info |
||||||||
Module | CSV Parser | Feeds | Import | Import HTML | Migrate | Node Import | Taxonomy CSV Import | User Import |
Maintained | No | Yes | Yes | Yes | Yes | Yes | Yes | Yes |
Releases (Status) | d6 (alpha) | d7 (alpha), d6 (beta) | d7 (dev), d6 (dev) | d7 (dev), d6 (stable), d5 (stable), d4.7 (dev) | d7 (stable), d6 (stable) | d6 (stable), d5 (stable, unsupported), d4.7 (stable, unsupported) | d7 (stable), d6 (stable), d5 (stable), d4.7 (stable) | d7 (stable), d6 (stable), d5 (stable), d4.7 (stable) |
Dependencies | FeedAPI (and Feed Element Mapper recommended) | ? | Drupal Core | / | Views, Table Wizard, Schema | Drupal Core, Date API | ? | / |
Target Audience | ? | ? | Developer | Site Admin | Developer | Site Admin | ? | Site Admin |
Documentation | README.txt | Drupal.org docs, screencasts | Example modules included (extensive) | Walkthrough and readme's included, Code commented verbosely | In progress | In progress | Help page | API, End User |
Weight/Drupal Core Upgrade Path | ? | ? | 91 lines of code. Easy to upgrade | Complex, but becoming more modular | Heavy | Complex but modular | ? | Medium (d6 up. testing time) |
Track Record | ? | ? | Proven against import of +1.7 million records | Works for me. Does often need template tweaking on a per-site basis. | Proven against import of +3 million users | ? | ? | Infinately scalable via cron runs. Tests via SimpleTest (d5) |
Functionality |
||||||||
Module | CSV Parser | Feeds | Import | Import HTML | Migrate | Node Import | Taxonomy CSV Import | User Import |
Interface | ? | API, UI | API | API (in dev), UI | API, UI in Migrate 1 (no UI currently, August 2011, in Migrate 2) | API, UI | ? | API, UI |
Input | A CSV file or a zipped CSV file | RSS, Atom, OPML, CSV (and XML, HTML with plugin) | Oracle, MSSQL, MySQL, any datasource PHP can communicate with / CSV, XML, HTML, images, video, flash ... any files | HTML site mirror, as through wget. / XML, XHTML, HTML, Also all resource files (images, attachments) into /files/* . | MySQL, CSV, TSV. Table Wizard hooks allow support of any file type. / Doable through hooks | CSV, TSV, ... (any delimiter separated values file) | CSV file (comma-separated values), or a copy-and-paste list | CSV |
Internet Imports | ? | ? | web pages, feeds ... anything accesible on the internet including other API's | Proof-of-concept (single page http import demo) available. Remote spidering, maybe one day. | Doable through hooks | No | ? | No |
Interoperability | ? | ? | Supports any module that extends nodes or users | Plugin framework available for extending node fields. Mostly undocumented. Support for third-party (CCK, nodewords etc) extensions available. | Supports CCK, Email Registration, Content Profile. Can support any module through hooks. | Can support any module through hooks. | ? | Can support any module through hooks. |
Processing (Batch, Cron, Single) | ? | ? | Batch API | d5: ran a Single process with timeout risk. d6: Batch API |
Cron, Single | d5: Single d6: Batch, Cron |
? | Cron, Single |
Output | Data into Drupal content types | Users, nodes, terms or simple database records | Nodes (any content type), Users, Roles, Taxonomy | Nodes (also additional node fields, images, attachments, downloads) | Nodes, Comments, Users, ... (hooks to define any destination) | Nodes, ... (Comments, Taxonomy, Users in development) | Terms in existing or new vocabularies | Users, Profiles, Nodeprofile, Organic Groups, ... (hooks to support anything else) |
Reporting | ? | ? | Pass/Fail stored in import tables for developer use | Mixed | Per-object errors/warnings/notices stored | Per-object errors/warnings/notices stored | ? | Itemised error report to user. |
Settings saved for reuse | ? | ? | ? | ? | ? | No | ? | Yes |
Import (Source-Specific)
Collaboration |
|||||||
---|---|---|---|---|---|---|---|
Module | Import Typepad | Joomla | Phorum | phpBB2Drupal | vBulletin to Drupal | Wordpress Import | WP2Drupal |
Project page links here | Yes | Yes | Yes | No | No | Yes | Yes |
Communicated with other maintainer(s) | ? | ? | ? | ? | ? | ? | ? |
General Info |
|||||||
Module | Import Typepad | Joomla | Phorum | phpBB2Drupal | vBulletin to Drupal | Wordpress Import | WP2Drupal |
Maintained | ? | ? | Yes | Yes | Yes | Yes | Yes |
Releases (Status) | d6 (stable), d5 (stable), d4.7 (dev) | d5 (stable) | d5 (dev) | d6 (stable) | d6 (rc), d5 (stable) | d6 (stable), d5 (stable) | d6 (stable) |
Documentation | ? | ? | Limited - README.txt | Handbook pages, README.txt, directions on screen | ? | Stable | WIP - readme.txt |
Functionality |
|||||||
Module | Import Typepad | Joomla | Phorum | phpBB2Drupal | vBulletin to Drupal | Wordpress Import | WP2Drupal |
Input | ? | ? | Phorum DB in MySQL | phpBB 2 or 3 | vBulletin forums and users | WordPress eXtended RSS (v2.1+) | WordPress DB in MySQL (v?) |
Processing (Batch, Cron, Single) | ? | ? | Cron | ? | ? | Single | Single |
Output | ? | ? | ? | Drupal equivalents to phpBB data: user profiles, containers, forums, topics, comments, attachments... | Drupal forums and users | ? | ? |
Reporting | ? | ? | Limited messages during import. | ? | ? | Limited, after import | Limited, during import |
Settings saved for reuse | ? | ? | ? | ? | ? | ? | ? |
Export
Collaboration |
||||||
---|---|---|---|---|---|---|
Module | HTML Export | Profile CSV | Save to FTP | Services | Views Bonus Pack | Views Datasource |
Project page links here | No | No (author disabled contact from g.d.o) | Yes | Yes | Yes | Yes |
Communicated with other maintainer(s) | ? | ? | Yes | ? | ? | ? |
General Info |
||||||
Module | HTML Export | Profile CSV | Save to FTP | Services | Views Bonus Pack | Views Datasource |
Maintained | Postponed; looking for co-maintainers | ? | Yes | ? | ? | ? |
Releases (Status) | d6 (stable), d5 (stable) | d6 (dev), d5 (stable) | d7 (stable), d6 (stable), d5 (stable) | ? | d6 (beta), d5 (alpha), d4.7 (stable) | d6 (alpha), d5 (alpha) |
Dependencies | ? | ? | Core path module, D7 requires Querypath | ? | ? | ? |
Target Audience | ? | ? | For site admins/users who want to manage content in Drupal, but display it as static HTML on another server | ? | ? | ? |
Documentation | Blog | ? | ? | ? | ? | ? |
Weight/Drupal Core Upgrade Path | ? | ? | ? | ? | ? | ? |
Track Record | ? | ? | ? | ? | ? | ? |
Functionality |
||||||
Module | HTML Export | Profile CSV | Save to FTP | Services | Views Bonus Pack | Views Datasource |
Interface | ? | ? | Node Edit page | ? | ? | ? |
Input | Drupal sites | Users, Profile data (may be deprecated in 6.x since Views can deal with profile data) | Node data | ? | ? | ? |
Processing (Batch, Cron, Single) | ? | ? | ? | ? | ? | ? |
Output | Static HTML files | CSV | HTML (via FTP) | XML | XML, CSV, TXT, Word.doc (all using included "export" package) | XML (raw), OPML, Atom, Simile/Exhibit JSON, Canonical JSON, JSONP/JSON in script, FOAF, SIOC, DOAP, hCard, hCalendar, Geo |
Reporting | ? | ? | Uses drupal_set_message for errors | ? | ? | ? |
Settings saved for reuse | ? | ? | Yes has administration options | ? | Saved with the view | ? |
Comments
Open Letter to Import Module Developers
It looks like the wiki is mostly complete and contains a lot of good information comparing a solid cross section of Import modules. Each of us have very valid approaches to handling imports and I don't want to detract from anyones efforts ... but there is an extremely valid argument that the Drupal community as a whole benefits from collaboration. I've been brainstorming with a few others about this issue and would like to reach out to you for your thoughts, opinions and interest in collaborating.
If you reflect interest we would like to provide an opportunity for us to get together and talk. Mike has a great group going at http://groups.drupal.org/migration-drupal that I think we could leverage and comments have been enabled on the wiki page at http://groups.drupal.org/node/21338 ... in addition to that I would like to propose a dual layered code sprint that happens the weekend of June 27th and 28th. We are having a Drupal Camp in Colorado with a capacity of up to 400 attendees. The most desireable outcome would be if each of you could make it to that event and we can come up with a way to make Drupal Imports rockk for everyone. The likelyhood of that happening in person may be a bit lofty so I would like to propose coupling the code sprint with a virtual code sprint similar to http://groups.drupal.org/node/18443
Following this approach would give us almost 2 months to first hash out amongst ourselves the direction we would like to take and organize community support.
I'd like to recieve your thoughts on this proposal before taking it further. Thanks so far for all the great work!
thoughts
Well first, I'm not going to be in person anywhere (unless I get sponsorship or something like I did last code sprint). :-/
However, where I'm at is:
- enough players have been sniffing around to push import_html forward, so the d6 will be out and about within a month.
Certain API bits are moving forward, and certain functions have dropped out to become modular (contrib or glue modules can catch the raw data and the newly formed node pre-save and add their own data extraction additions via hooks)
I've watched the birth and death of import/export API (last activity April 2007) and am wary of looking for the one true answer. (I DID think it was possible in the beginning)
Migrating from different datasources - which is pretty much what the tools in this realm do - are unique jobs.
I've looked hard at my own and others code and I've not seen the point at which these jobs hit common ground. Node import is a different tool, and I use it myself, learned from it, and even patched it a bit. It's the closest to import_html of the lot - but I've not yet found a function that could be placed in common.
Maybe others can see some points of commonality. I'd be interested to hear. My docs are verbose on methodology and a walkthrough illustrates what can be done.
FYI, here's a slideshow of someones import experiments.
http://www.slideshare.net/emmajane/moving-in-how-to-port-your-content-fr...
.dan.
Commonality
The common portion of pretty much all import modules is the actual import. From what I looked at and from my experiences writing a custom wordpress import module (old wp version), saving node, term, comment, and possibly other content data should be the same. It's the data source and data mapping that's different.
It also breaks apart with users. Since importing users from a plethora of different encryption types is impossible to transfer passwords over unless they're plaintext.
Different object types
Different object types (nodes vs users) have different needs.
But at the bare metal, all the node import routines have just one section in common :
<?php
$new_node = (object)array(
'title' => $new_title,
'body' => $new_body,
'type' => $new_type,
// plus some housekeeping like $user
);
node_save($new_node);
?>
It's getting it to the point where you can plug the values into a node object that is the hard bit, and that's highly individual.
We MAY be able to find some useful common routines or CRUD to assist with creating some bits (I needed a bit of extra work to be able to create arbirtrary menus on the fly, for example) but the point we truly have in common is already API. node_save()
... Of course, this is because I look at the entire task from a Drupal API POV. I've seen lots of other scripts that try to do their import via direct database updates. Ug. Those guys may have different things in common, but I'm not in that camp.
Different commonalities
Really? Because the commonalities that I've become used to are different:
You need to decide which data you need to import and from where.
You need to decide how to handle reporting successes and failures, and what to do with them.
You need to set up a process to do the work in slices, because importing 50,000 nodes in one php request usually fails - happily, the batch api helps a lot with that these days.
These are all tasks which you have to do on every import, and having code to support and make these tasks easier is useful.
-john
-john
Thanks, I agree with the
Thanks, I agree with the second two, but the first doesn't seem to be a commonality. Or I guess code-wise the way that migrate, wordpress import, and user_import aren't common imo. I guess abstractly it's all common: find data, get data into proper array/object data, save data.
A handler class system would be nice.
I really need to look at import module. I was excited when I first saw it on the session list at DCDC, but it was dropped. I haven't had the time to browse through that api yet.
Regarding the report of the demise of Import/Export API
As of the past few days, things are looking promising for a 6.x release of the Import/Export API some time soon. Some people are reporting success already and a beta quality release sounds like it's not far off.
We've had a good amount of
We've had a good amount of email communication between the authors of the WP2Drupal, Import/Export API, Migrate, and Import API modules. It sounds like the desire to start unifying our efforts is there ... now we just need to figure out what we will be doing and how. As expected, the idea of a code sprint where we can all collaborate is unlikely but we can still move forward. I understand that we all know our own modules quite well, but in an effort to fully understand the situation we need to take the time to evaluate the other modules in this category. The Migrate modules maintainer came up with these questions and I think they are a good starting point ... I'll be posting my answers after testing the modules listed above:
"As a starting point, I think an interesting exercise would be for each of us to look over the various modules in the same category, then write up:
If we can first converge on the unique value of each of these modules, we can move from there to figuring out how to make them work together..."
The stages of migration
Sorry I've been quiet here - I was off on vacation last week, and I'm slowly getting caught up. I haven't had time yet to look closely at the other modules. But, some quick thoughts on the prospects of a common framework...
My experience doing large-scale migrations - not just with Drupal, but also with a big ERP (payroll/HR) migration a couple of years ago - is that the full process needs to look something like:
Obviously, not all stages need be implemented in a single module. But, by separating input (Analysis and Consolidation) from output (Import, Auditing), I do think one can have a common framework, dealing with various sources (HTML, Wordpress, etc.) independently (more-or-less) from destinations (nodes, users, etc.).
Mike Ryan
http://mikeryan.name/
Mike Ryan
I agree with Mikes analysis
I agree with Mikes analysis of the phases - it's certainly the way I go about it.
Steps 1-2 is the hard part - and is the bit that's different for everybody.
There may be some crossover we can do between the approaches for step 3 however. A queue of pending items, preview and rollback.
FWIW, my 'common form' for import_html is vanilla XHTML with a bunch of semantic meta tags and a few microformats. Once I've got from 'clunky data' to standardized clean data, I know the import will go smooth.
There is a second 'common form' inside step 3 - the Drupal PHP object itself - which is what some import mechanisms go straight to. I find that a bit too close to the bone and (although I do it) putting too much care into adding elements to $node always feels like I'm bypassing the API and almost touching the DB directly.
Maybe we just need better CRUD tools.
With taxonomy_xml, (import/export taxonomy terms) I broke the different formats up into their own libraries - which scan the input and return a queue of almost-complete Drupal $term objects. For that one, queuing and partial imports was harder as I needed internal references most of the time. I needed to save a term to get its ID so that another term could point to it as parent etc.
Joining
Just thought I'd say hello :)
I've been looking for an area to get involved with Drupal for a while, and seeing as I wrote the ImpEx for vBulletin (and 120 migration systems) this might be a good place to start !
Just reading over what is going on and getting up to speed, so I can lend a hand with some thoughts.
Cheers,
Jerry
http://drupal.org/project/mig
http://drupal.org/project/migrator (in development)
http://drupal.org/project/transformations
http://drupal.org/project/transformations_drupal
http://drupal.org/project/transformations_csv
http://drupal.org/project/fieldtool
http://drupal.org/project/deploy
Export:
http://drupal.org/project/export_docbook
Daniel F. Kudwien
unleashed mind
Daniel F. Kudwien
netzstrategen
Righty looks like a
Righty looks like a combination of transformations sitting on top of the Import / Export API.
Middleware
See also: http://drupal.org/project/middleware
Modules table overhaul
Hi all, been using Drupal since 4.6, used to make some small contributions to d.o here and there quite some time ago (satori at abc3400 dot be).
Anyways, I've done a major edit of the modules tables. Apparently, there's no way to view the revisions nor the logs, so I'm reposting my log message here for posterity:
Peace
I've been evaluating all of
I've been evaluating all of this information for a long time and have come to the conclusion that it would be in the best interest of the Drupal community to keep efforts focused on the best solution. For that reason I've discontinued the Import module and placed a reference on it's project page to the Migrate module. I still don't think the Migrate module is where it needs to be but it is leaps and bounds above the other modules in regards to planning and approach. I'll be using Migrate with all of my upcoming projects and look forward to submitting patches.
New module
Hi, I'm currently developing a Drupal to Drupal synchronization/migration module, should it stands in this list?
See it there http://drupal.org/project/yamm
Pierre.
Pierre.
My brain literally just
My brain literally just popped from the weight of my jaw hanging open in disbelief.
@pounard - Please add it -
@pounard - Please add it - it looks very promising!
@cyberswat - Thanks for that note. I'm sorry that I never did do a really thorough review of all the modules covered here, although I did take a quick look at each of them. I'm not convinced there is a "single solution" - there is room for different domains (Drupal-to-Drupal migration/synchronization vs. import from external CMSs, not to mention the export side), and different approaches (quick-and-simple vs. kitchen-sink). I do feel Migrate is the best available external-CMS/kitchen-sink solution (naturally, since I've built it to meet my needs), but there's room for other modules.
I think it is still very valuable to maintain comparisons among modules in this general space, both to help people find the best module for their particular situation and to help developers avoid reinventing the wheel. Let's keep it up - but let's talk about the scalability of the chart, which is difficult to manage - can we do a Google spreadsheet or something like that?
Kevin, I'm looking forward to seeing your improvements to Migrate.
Thanks.
Mike Ryan
http://mikeryan.name/
Mike Ryan
Yamm module added to matrix
@mikeryan did it.
@all My module is more like a Drupal to Drupal mass synchronization module than really an import/export module, I think some topics on general table are not really revelant for this module.
EDIT: typo
Pierre.
Pierre.
Note on this wiki page
On small screens, blocks cover the right side of the matrix. This is quite annoying to fully read it I had to use firebug to put the whole blocks part as "display: none;", not much elegant.
Pierre.
Pierre.
New table: "Import and export"
Pierre wrote:
I've moved the modules that import and export, from "Import (Generic)" to their own new table, "Import and export". This was necessary, I think, and I've made it in two steps to verify it carefully. As a secondary effect, this wiki page is more readable now.
Node Export
http://drupal.org/project/node_export It's not in the table now, and I just used it yesterday and found it pretty helpful for Drupal-to-Drupal data migration. I'll try to add it in, although I'm not associated with the project at all. (It turned out to be the best solution for moving a webform from one site to another.)
Node Export and other modules added
I've just added a column for Node Export with a few details (and ordered the table columns alphabetically). Others may complete the data.
Also, I've added some of the most used modules in this field, according to the drupal.org's list of import/export modules (link ordered by usage stats).
This is a big chart and I'm
This is a big chart and I'm not seeing one of the main deciding factors -- can import profiles be saved?
Eg. user import and backup and migrate - both permit saved profiles, which allows a site dev to preconfigure some and then pass the routine along to a site admin and it covers a wide range of abilities of that site-admin. I can't pass along 'node import' with the same ease at all.
This variable is perhaps more meaningful than 'target audience'
How about 'saved import profiles'?
New rows: "Settings saved for reuse"
I've added those requested new rows (just the html, and few details, because of lack of time), to facilitate that others can add the data. Since "profiles" could be confused with user profiles, the new rows are labeled "Settings saved for reuse".
sure - sounds reasonable
Thanks. And sure, that sounds reasonable.
Fits in with the language of the user_import module - http://drupal.org/node/137672
While it's essentially the same routine, backup_migrate uses the variable $profiles and $profile_id in a file named profile.inc , which adds a bit of confusion into the mix
Case study with Migrate module
The recent case study on the front page of drupal.org, Trócaire: Working for a Just World, says:
"we used Migrate to pull in the data from the existing CMS"
(...)
"The decision was taken to switch to Drupal, but there was still the issue of migrating all the existing content, files and users to Drupal. Using the excellent Migrate and Table Wizard modules, created by Mike Ryan and Moshe Weitzman of Cyrve, this task was made much easier. Over 2000 pages, 500+ files, almost 600 taxonomy terms and close to 100 users were successfully migrated across."
(...)
"Migrate, Table Wizard and Schema - allowed relatively pain-free migration of all existing content and users to Drupal."
(...)
"Trócaire were happy to contribute back all custom code and patches developed during the project to Drupal. Here are some of the main ones:"
(...)
"Migrate - #484404, #551254, #541700, #540908, #411156, #525996, #426824 and #520264."
Import Module "Mashup" Wizard proposal
I've got an "Import Wizard" module I've been developing for my own use which essentially allows an administrator to create a profile which is a "mashup" of several of the different modules mentioned on this page. My proposal: http://groups.drupal.org/node/37106
Migration tutorials
[#528726] also has some links to good tutorials for migrating/importing content.
Frank
My LinkedIn profile
SO what module to use when 1)
SO what module to use when 1) Upgrading data from D5 to D6 and later on to D7? and what module to use to only import-export nodes and taxonomy? The list is long..and still not having the right module for both situations..sorry if misplaced remark here..
greetings, Martijn
Add Feeds module to this list
http://drupal.org/project/feeds promises to be a powerful way to import data....
HIGH PRIORITY: Interoperability - Node import and Feeds modules
Interoperability of Node import and Feeds module would be much appreciated. Currently, the Feed URL field -- related to the Feeds module -- is not exposed (i.e., not supported) in the Node import module for field mapping purposes.
Node import module
http://drupal.org/project/node_import
Feeds module
http://drupal.org/project/feeds
(Node import is flexible, can be extended to support any module through hooks.)
Marcus Ledergerber
Re: Add Feeds module to this list
I've just added a column for Feeds module to Import (Generic), and updated version and UI data for Migrate. Currently, both Feeds and Migrate seem to be among the most popular import modules.
These comments are in the
These comments are in the context of Drupal->Drupal export/import only.
On the Migrate page mikeryan says:
The ideal place to implement migration support for a contributed module is in that module, not in the Migrate module.
I would go just a bit further:
The only place to implement migration support for ANY module is in that module, not in the Migrate module.
If every module that persisted data on a site provided hooks to:
All Migrate would have to do is orchestrate these processes and determine how the array data is stored. It requires no knowledge of the inner workings of any module and is completely generalised and extensible.
cheers
Another one for - Import and Export (Drupal to Drupal)
Another module that should be in the "Import and Export (Drupal to Drupal)" section is http://drupal.org/project/content_distribution
Agileware are an Australian Web Team specialising in Drupal, WordPress and CiviCRM.
Agileware provide development, support and hosting services.
https://agileware.com.au
Importing...
I've been surprised at how difficult it is to find a working module to import in a csv of data to become nodes. The closest I could find is node_import, but anything that is a checkbox, yes/no, etc. gives a unicode error no matter what you do. And I've been able to get zero assistance from the module's maintainer(s) regarding the issue. I'm not the only one having it - others are as well.
It doesn't seem like it should be that hard to have a node import module that can at least work with the basic cck and Drupal fields.
Jenni Simonis
http://www.forwardsupport.com
Feeds?
Have you tried feeds? I was using node_import last week but replacing it for feeds as seems better, not sure about checkboxes though...
Looked at it...
I looked at it, but I couldn't find anything in it that allowed me to import checkboxes, radios, or selects (I forgot that in the listing). Much of our textboxes are content taxonomy, but not all of it.
Jenni Simonis
http://www.forwardsupport.com
Feed
I am trying out feed, but it seems there is a problem with importing the node title. Did you run into this? Multiple people have reported this and it keeps getting marked as fixed when in fact nothing has been fixed.
Jenni Simonis
http://www.forwardsupport.com
Deploy & revert
It may also be worth mentioning UUID Features Integration, and Default Content modules – both allowing content to be features exportable.
Worth noting: the latter has additional functionality that removes the content once the feature is disabled. However the module may be discontinued at some point in favor of uuid_features (perhaps this extra functionality could be added as an option to uuid_features).
Dev/Staging/Live deployment is a big problem
Dev/Staging/Live deployment is a big problem.
This seems to have been worked on again and again and seems to be a difficult one to crack.
I have helped set up a dev/beta environment which works will for a team of devs - we can all work together on developing a new beta version of a site.
The next stage is pulling the live data from the live site to the beta to create the fully populated new site. I've made an initial module which can export and then import data (nodes, taxonomy terms). The important point here is this - in the docs with the module I explain how a full dev/demo/live staging set up can work.
The module is at:
http://drupal.org/sandbox/bailey86/1278830
Any comments suggestions gratefully received.
@Bailey86
I am struggling with this dev/staging/deployment issue ( http://groups.drupal.org/node/207443 ) ...
Sure will check out your contribution today, but since you're clearly quite experienced in that field already, I thought may be you could help me better understand the ins/outs and available solutions as of today ...
Many thanks :-)
@snoopy
Put simply...
You have a beta copy of the site - this must be respected and kept in good order. Developers should test out any changes thoroughly on dev copies before applying any code changes to the beta site. Nobody owns the beta site - it is shared between devs.
(We have scripts which will make an exact copy of beta on to a dev's dev copy in about two minutes - so they can really try out and test out their changes. Once they know they can apply changes OK they then block beta updates by anybody else (we request to hold a 'virtual spoon'), copy out the latest beta copy via the script, make their changes and then use another script to copy their dev copy up to the beta site. Since the script uses code repositories and makes backup copies before any actions etc it keeps the process safe(r)).
Add to this that the scripts use code repositories to keep track of the code and devs are expected to update a wiki with their changes and the revision number etc.
The advantage is that site users/admin can be testing things out on the beta copy and adding content and test content etc etc.
OK - so the beta site is running OK - and being developed nicely - and being checked out/tested by the stakeholders.
At some point you will want this to become the new live copy of the site.
The data_export_import module - http://drupal.org/sandbox/bailey86/1278830 - can be installed on to the live site. And this will enable you to export the data - nodes, taxonomy terms (and soon users) - to files. (Other data can be exported if you need by the creation of additional profiles). You can then install the data_export_import module on to the beta site to import these data files into your beta copy. Various checks are built in to make sure vocabularies exist and content types match etc. (more checks might be needed).
The important point is that the data gets imported with the same ID numbers - so all links work as expected - also, any (test) data which has been created on the beta site but which is not in the data file will be deleted. (Think in terms of rsync with the --delete option).
When this all imports nicely the beta site then becomes the new live site.
A Brucie bonus is that is there turns out to be problems with the new beta site then you still have the old live site running to fall back to.
import from excel to a field
Any module for importing from excel?
Importing - the bones of the matter
I have been around the periphery of drupal since 2010, and still am not a master of the universe, but I do like learning and trying things out. I not even competent in drupal and still struggling with some of the basic concepts. So I have had websites since the 1990's of one form or another. Mostly HTML 3,4 and 5, have not done much with php or the other higher level languages, was very involved on machine code basic and the like.
So we create a basic drupal system and we do not want to go and recreate pages which are already in existence but want to store them in the drupal system and realize we have adapt the pages create the taxonomy links etc. So what do I need to do?
It seems to me if I can import them into nodes (the storage points) and know where the are I can the name the place them link them etc.
What do I need to know and be able to do?
I need to import them and know where they, I need a little document to explain how to get them into the database then to manipulate them.
From the excellent brief above it seems the is a module called HTML import so I shall go away and try that and hope it works well with Drupal 8x