This week at DrupalCon Denver, we held a BOF to discuss the proposal at http://drupal.org/node/1052692 to incorporate some form of the Migrate module into core in Drupal 8 to handle major version upgrades.
Migrate module roadmap
My plan is for Migrate 2.4 to be the last functional release of Migrate V2, with improving the file destination and field handlers being the main theme. Migrate V2 would then go into maintenance mode (bug fixes only) while we focus 99% on D8, possibly backporting that core API (plus contrib enhancements) to D7 as Migrate V3 (depending on demand, and availability of underlying APIs in D7 similar to the dbtng and autoload modules in D6).
Migrate features to go into core
- Management of map tables (tracking source->destination relationships).
- Source/destination abstractions.
- Handlers/listeners - ability to manipulate the data at key points in its flow from source to destination (equivalents of prepare, prepareRow, complete, callbacks...).
- Handling of simple relationships - stubs, sourceMigration.
- Dependency handling
- Highwater marks
- Dynamic migrations (may actually become the default, or even only mode)
- Simple UI for performing migrations (upgrades) - two buttons, import and rollback.
- Support for PDO and XML sources.
- Example module supporting automated tests (as wine.inc in migrate_example does now).
Migrate features not to go into core
- Rest of migrate_ui.
- Field mapping annotations (description, issueGroup, etc.).
- systemOfRecord = DESTINATION (although this may be difficult to add at the contrib level - we need to be sure the core API gives us the right entry points to be able to do this).
- Drush commands (well, maybe)
- Support for Oracle and MS SQL sources.
- Support for CSV and JSON sources (or should one or both of these go in core?).
- Examples beyond any that support tests.
Improvements to make over contrib Migrate
- Smarter sources - Right now, the source and destination classes aren't quite mirror images. Sources operate at the data layer - SQL, XML, CSV - while destinations are at the next layer of abstraction - nodes, users, terms. We would like an infrastructure that supports mapping a Drupal 7 node (source) to a Drupal 8 node (destination). How to deal with either a direct DB connection or XML feed as the source of those Drupal 7 nodes?
- Better handling of complex relationships and structured data, like collection fields.
- Destination support for any core data not currently covered by the contrib module.
Support for D2D upgrade path
- We need to migrate configuration (including content type and field definitions) - so, a Drupal variable source and a D8 config file destination.
- DX is critical - contrib modules will need to implement this API for their upgrades, we don't want this to be an impediment to contributors.
- Is Batch API adequate for big migration/upgrade jobs? Short of including drush with core, what else can we do? Recommend using the contrib module with drush support? Or, can we implement the drush commands in core?
- This should support first PDO databases as a source, then the output from the Web Services Initiative, then more general sources (e.g., aggregrator/services-based feeds from pre-8 versions of Drupal).
I see three pieces to be committed to core:
- Import API - the general purpose get-stuff-into-Drupal classes.
- Upgrade API - Should be a dirt-simple DX for not only core components but contrib modules to transform data from another Drupal installation into Drupal 8. From any version of Drupal, including 8, so "Upgrade API" may not be the best name for this.
- Core upgrade support - implementations of the Upgrade API to pull data from other Drupal installs. First, of course, would be Drupal 7 as a source - it would be great to also support Drupal 6 and Drupal 8 as sources, with contrib support for earlier Drupal versions.
So, let's start figuring out the architecture of the Import and Upgrade APIs. I will add separate posts to the group for those.
Thanks to everyone who came to the BOF and contributed to the above:
Mike Ryan (mikeryan)
Andrew Morton (drewish)
Ashok Modi (btmash)
Adrian Rollett (acrollet)
Camilla Krag Jensen (naxoc)
Joe Stewart (joestewart)