Interested in helping test Drupal 6 to Drupal 8 migrations? Start here for instructions on how to get set up, and where to start testing. Find us on IRC in #drupal-migrate.
Weekly Meeting
We meet weekly on Thursdays at 1:00 UTC (Wed 6pm PDT, Wed 9pm EDT, Thurs 9am benjy-time). Post a comment on the event page if you'd like to participate (and not just watch) and we'll be sure to invite you when the hangout begins.
Current Priorities
D6 > D8
All issues are in the core queue.
Pick any active issue, read the instructions, assign to yourself, and hack away! If you can't complete it, post a partial patch and unassign.
- Finish the remaining tests (intermediate skill)
D7 > D8
All issues are in the core queue.
About IMP
WHAT: Import API in Drupal 8 core, based on the Migrate project. Also in core: Drupal 6 and 7 import using this API based on the Drupal-to-Drupal data migration project. Read and comment on Mike Ryan's thoughts at http://goo.gl/WnTOB0.
WHERE: If you'd like to be involved now, find us in IRC on #drupal-migrate. If you have any doubts about what to do, find us in IRC or post in the issue! Work takes place here: https://drupal.org/sandbox/chx/2105305. The UI is at https://drupal.org/sandbox/mikeryan/2271813 .
Migrate API documentation can be found at https://drupal.org/node/2127611
WHY: So that Drupal 6 sites can move directly to Drupal 8. (And because the current upgrade path is unmaintainable.)
HOW: We're currently doing targeted testing and patching of the D6->D8 migration path. Want to get involved - introduce yourself in #drupal-migrate on IRC!
Now broadcasting weekly meetings on Google+
How to test Drupal 6 to Drupal 8 migrations
Interested in helping out with the Migrate in Core initiative? This page is a great place to start, as to ensure that you've got everything you need in-place on your local machine.
Prerequisites
- Comfort with the command-line - The user interface at this time is drush, so some command-line comfort is required.
- PHP 5.4 - required for Drupal 8.
- Git - https://drupal.org/documentation/git/install
- Drush 7 - https://github.com/drush-ops/drush#install---composer - for manual testing of individual D6->D8 migrations, install the latest version of Drush. It is strongly recommended to install Drush 7.x via Composer.
- A Drupal 6 site - Migrate only reads from this database. Even so, it is not recommended to run migrations against a production database! For initial testing, use a fresh Drupal 6 site with only the CCK and Link modules enabled and some sample content.
- A Drupal 8 site - It must be the latest version, https://drupal.org/project/drupal/git-instructions
- Enable the migrate_drupal module, this enables the migrate module as well.
drush en -y migrate_drupal - Make sure drush can write to the config directory.
sudo chmod -R a+w sites/default/files/config_*/active
- Enable the migrate_drupal module, this enables the migrate module as well.
- Test your environment - Run a block migration. If everything works, then you're ready to hit the issue queue!
IMP work returns to the sandbox
Migration in core, aka IMP, returned to working in the sandbox last week. This is a slightly unusual development so some explanation is in order. The fundamental API and several migrations have been committed to core, establishing that this initiative actually has legs and will happen, but now that that's in place, it makes sense to return day-to-day work to the sandbox because:
- Core committers are (rightly) focused on beta blockers / criticals, so these patches sit around for a while before getting committed
- When the patches sit, they block other migration work
- The API is still young and we could use space and time to flesh it out more before submitting to core where it becomes more "locked down"
- Arguably, the patches will be easier to review as a whole, because then we can test the D6 -> D8 migration itself, and not just piecemeal parts of it.
We will continue to use the core queue for:
- Major adjustments to the migrate API already committed to core
- Bug fixes to the core API
If you have questions, post them to this discussion or find us in IRC in #drupal-contribute.
Read moreDrupal 7 updates that need Drupal 8 migrations
Since we decided to remove support for the Drupal 7 to Drupal 8 upgrade path, update hooks are no longer being added consistently and it no longer makes sense to add update hooks that will never be run. This document should be used to document things that would have required update hooks and may now need data migrations.
See https://drupal.org/node/2168011 for more information.
Variables to state or config
(list issues here with a brief note)
Added or removed database fields and tables
Not all schema changes require data migration.
Read moreDecember IMP update
The December holiday season is traditionally a little slower than the rest of the year, but it didn’t stand in the way of steady progress for IMP!
Major milestone achievement/s:
The first complete config entity migration, #2150407 Migrate D6 roles and permissions, was committed to core (Thanks, alexpott!). This patch introduced the entity destination in a very minimal state, the first two process-specific plugins, and a unit-tested source plug in as well.
Steady progress:
Read moreWriting D6 to D8 entity migrations
Some part of this page if out of date, mainly because of Migrations are plugins instead of configuration entities. This page was last updated in Feb 2015.
Setting up
You’ll need clones of Drupal 6, 7, and 8. They don’t have to be installed, just cloned.
D6: The install file so you can look at hook_schema implementation
D7: The *.install files can help you locate more information about what changed between 6 and 8 in the update functions.
D8: The entity class
IMP-specific PHPUnit Directions
We're collecting questions from today's sprint in hopes of getting documentation together for future work.
Where do I put my tests?
Add tests to the exisiting test files.
For MigrateExecutable.php, add to:
- core/modules/migrate/tests/Drupal/migrate/Test/MigrateExecutableTest.php
For Sql.php, add to
- ./core/modules/migrate/tests/Drupal/migrate/Tests/MigrateSqlIdMapTest.php
What does the getMock function do? What's the high-level idea of the function and how do I know what to pass to it?
Read moreSetting up to contribute to IMP
Getting set up to contribute to IMP
- Visit the IMP sandbox Version control tab and follow the instructions to clone the project. The sandbox contains both Drupal 8 core and the migrate module so it takes a bit to download.
- Work for this project is located in
core/modules/migrate. If you have access to commit directly to IMP, commit directly to this 8.x branch. Otherwise, create and submit patches to the IMP issue queue. Directions are on the Version control tab under the heading Patches. Note: chx will handle consolidating work into patches for core as necessary. - It is helpful to download and install Drupal 6 in a separate directory so you can test enabling modules, configuring them, creating content and seeing how that content and configurations are stored in the D6 database.
High-level IMP architecture
Backing up a bit to a high-level view of the architecture, right now we've got the whole thing in a single Migrate module. I'd like to see some separation between the APIs that will service all kinds of migrations, and the support specific to Drupal-to-Drupal upgrades, as well as identifying what (if anything) requires a module. The buckets of code I picture:
Core
- Migrate API: Most of what we have now.
- Upgrade API: Any classes specific to Drupal-to-Drupal migration.
- Upgrade module: D6/D7 migration configurations, plus UI.
Contrib
- migrate_plus module (or even call it "migrate", if there's no module in core?): General migration UI, D7 migrate features not going into core such as different source plugins, etc.
- migrate_d2d module: Support for D5 and D8 sources, UI for remapping content types (i.e., support for custom Drupal-to-Drupal migrations as opposed to upgrade-all-the-things), etc.
Thoughts?
Read moreSecond IMP Sprint
The IMP team is working hard to complete a core migration path from Drupal 6 to Drupal 8 by the end of January 2014. Join us on November 19 between 9:00am - 5:00pm Pacific for our second sprint.
The IMP initiaitve lead, chx, will be guiding developers from all over via IRC in freenode's #drupal-contribute channel.
Portland-based OpenSourcery has dedicated a team of three outstanding developers to support a face-to-face sprint in downtown Portland, Oregon. If you're in town, consider joining us! https://groups.drupal.org/node/368358
Sprint Goal
The sprint goal is complete unit test coverage for
- lib/Drupal/migrate/MigrateExecutable.php
- lib/Drupal/migrate/Plugin/migrate/id_map/Sql.php
It's a great chance to get involved with Drupal 8 and get up to speed with PHPUnit in core.
Current tasks
- Among many things, chx is working on the FakeInsert, FakeUpdate, and FakeMerge classes so SQL writes in the id_map can be unit tested.
- Others are creating a slew of migrations and integration tests for variables-to-config. Detailed instructions are available, so you can just grab an issue without much previous knowledge. Reviews are happening swiftly (less than 24 hours) and chx and others are highly available in IRC to answer question.
- Late next week, we'll set up the sprint-day issues. If you'd like to lend a hand with that, let me know in the sprint planning issue.
PrepareRow gets a hook
filter_update_7003 is reading settings from a variable for each format it knows about. How do we handle this? Obviously we can cook this into prepareRow of the source class; but then how will contrib get on board? Reminder: we plan to use the entity load hook to let contrib extend the (module,delta) -> map mapping. It would make quite some sense to use the same hook as the event contribs will need to fire on but I am not sure what exactly they would change! Probably we need to provide some sort of extensibility for prepareRow apparently just not sure how. Another type of plugin??
Agenda for the 7/11 call
Where we are:
- The process syntax has been revamped completely, the new get has a test and there's documentation
- There's an entity handler but completely untested (as in I never ran it)
- There's a map process plugin but it's also quite untested
- bdone has written a few variables-to-config migrations (thanks!) others haven't, we are badly behind on them
Next steps:
- Reroll core patch
- In import, handle the case where $destination->import succeeds but there are no ids -- as in config
- Write more tests: migration and map process plugins, sql idmap. sql idmap tests should go into core patch #1
- Write the filter formats migration
- Create core patch #2 with the map process plugin, the filter formats source, the entity destination the filter formats migration and make sure all of this is tested
- Get people to write the variables-to-config migrations somehow.
- Entity destinations need the stub functionality. Not urgent.
Webinar: How to Migrate from .NET to Drupal – November 13@ 1:00 PM EST
Join Acquia and DOOR3 for an informative webinar on why migrating to Drupal CMS provides the flexibility and ROI your organization desires. With a strong practice around both .NET and open source (and Drupal), we understand all of the pros, cons and realities when it comes to CMS migrations. You'll hear an informative discussion on why this is the last move you’ll ever have to make.
Read moreLet's pivot
Currently the process values are a list of plugins and their configurations. This is causing a lot of headaches because of ordering. And also, noone thinks in terms in plugins. The Drupal 7 version of addFieldMapping was also taking destination as the first argument. Reading the current documentation at https://drupal.org/node/2127611 first might be helpful.
Read moreFirst core patch submitted
It's at https://drupal.org/node/2125717 . Focus is not the issue. Work continues as normal in the sandbox, next step is https://drupal.org/node/2125745: Variables to config
Read morePre-notes for the call on Oct 31th
As drupal.org will be down the document is copied to https://docs.google.com/document/d/1ubNnzQf38jp_yPKixg-x1DeiVoRvow9dLIqY... and the code is pushed to http://github.com/chx/imp
- Almost all of the D6 sources are done
- marvil07 is leading an effort to add requirements checking, in the beginning they are just modules existing and being the right schema version
- There's a new Source class, it's not pluggable, it wraps the source plugin.Same code as in SourceBase just separated out.
- The message display is in a separate class, it even has an interface, mandatory to pass to MigrateExecutable. Default implementation is just drupal_set_message().
- I am working on the first migration, id_map needs to in a semi decent shape.
What needs to be done:
- Review the current state with core in mind. Add missing doxygens.
- Once the first variable-to-config migration is done, we need to get another 50 of those
- Entity destinations. The base should be blatantly simple.
- url_alias destination.
IMP progress report #1: Past BADcamp, before core patch
So far we have established the structure of the new migrate module, there is a migration configuration entity which holds the configuration of the various plugins: source, process, destination and id map. As Views does, migrate also will have a MigrateExecutable class, responsible for "turning the gears" -- calling the proper plugins. There is a standalone, non-pluggable Row class which mostly just holds the source and destination data and does very little logic.
Versions and requirements
We need to version the sources for sure. I can't remember a case when a core minor release would break our queries but such things definitely happened in contrib (look at CCK it keeps the table names in a function because of this).
Read moreSourceBase split? Edit: yes.
When I look at SourceBase, it looks like a standalone class to me which wraps an iterator from the real source base. Should we split it off instead of having these funny named methods like performRewind, getCurrentRow? Our database classes already return an iterator, it's trivial to wrap reading a file in an iterator ( https://github.com/weppos/fileiterator/blob/master/FileIterator.class.php ) so I think this would be beneficial. Also we could skip computeCount and just use count() and make the sources implement Countable.
Read moreOn canonical representation
Yesterday jessehs was writing the filter formats plugin and we had a discussion live and then on IRC with marvil07 whether we should migrate the filter format table and then run a migration for each format to bring the filters over or just add the filter list to the formats in prepareRow. We are doing the latter.
Noone suggests bringing over entities and then fields separately either.
Read moreThe UI
As far as I am concerned, the UI will contain:
- the database credential fields straight out of the installer
- Edit:
A Drupal version select boxit seems we can divine this from the system table. - A file path textfield. (MIght need two for private files in D7)
- A button saying Migrate configuration
- A button saying Migrate content
