IMP

Interested in helping test Drupal 6 to Drupal 8 migrations? Start here for instructions on how to get set up, and where to start testing. Find us on IRC in #drupal-migrate.

Weekly Meeting

We meet weekly on Thursdays at 1:00 UTC (Wed 6pm PDT, Wed 9pm EDT, Thurs 9am benjy-time). Post a comment on the event page if you'd like to participate (and not just watch) and we'll be sure to invite you when the hangout begins.

Current Priorities

D6 > D8

All issues are in the core queue.

Pick any active issue, read the instructions, assign to yourself, and hack away! If you can't complete it, post a partial patch and unassign.

D7 > D8

All issues are in the core queue.

About IMP

WHAT: Import API in Drupal 8 core, based on the Migrate project. Also in core: Drupal 6 and 7 import using this API based on the Drupal-to-Drupal data migration project. Read and comment on Mike Ryan's thoughts at http://goo.gl/WnTOB0.

WHERE: If you'd like to be involved now, find us in IRC on #drupal-migrate. If you have any doubts about what to do, find us in IRC or post in the issue! Work takes place here: https://drupal.org/sandbox/chx/2105305. The UI is at https://drupal.org/sandbox/mikeryan/2271813 .

Migrate API documentation can be found at https://drupal.org/node/2127611

WHY: So that Drupal 6 sites can move directly to Drupal 8. (And because the current upgrade path is unmaintainable.)

HOW: We're currently doing targeted testing and patching of the D6->D8 migration path. Want to get involved - introduce yourself in #drupal-migrate on IRC!

Now broadcasting weekly meetings on Google+

eliza411's picture

IMP work returns to the sandbox

Migration in core, aka IMP, returned to working in the sandbox last week. This is a slightly unusual development so some explanation is in order. The fundamental API and several migrations have been committed to core, establishing that this initiative actually has legs and will happen, but now that that's in place, it makes sense to return day-to-day work to the sandbox because:

  • Core committers are (rightly) focused on beta blockers / criticals, so these patches sit around for a while before getting committed
  • When the patches sit, they block other migration work
  • The API is still young and we could use space and time to flesh it out more before submitting to core where it becomes more "locked down"
  • Arguably, the patches will be easier to review as a whole, because then we can test the D6 -> D8 migration itself, and not just piecemeal parts of it.

We will continue to use the core queue for:

  • Major adjustments to the migrate API already committed to core
  • Bug fixes to the core API

If you have questions, post them to this discussion or find us in IRC in #drupal-contribute.

Read more

Drupal 7 updates that need Drupal 8 migrations

Since we decided to remove support for the Drupal 7 to Drupal 8 upgrade path, update hooks are no longer being added consistently and it no longer makes sense to add update hooks that will never be run. This document should be used to document things that would have required update hooks and may now need data migrations.

See https://drupal.org/node/2168011 for more information.

Variables to state or config

(list issues here with a brief note)

Added or removed database fields and tables

Not all schema changes require data migration.

Read more
eliza411's picture

December IMP update

The December holiday season is traditionally a little slower than the rest of the year, but it didn’t stand in the way of steady progress for IMP!

Major milestone achievement/s:

The first complete config entity migration, #2150407 Migrate D6 roles and permissions, was committed to core (Thanks, alexpott!). This patch introduced the entity destination in a very minimal state, the first two process-specific plugins, and a unit-tested source plug in as well.

Steady progress:

Read more

Writing D6 to D8 entity migrations

Setting up

You’ll need clones of Drupal 6, 7, and 8. They don’t have to be installed, just cloned.

D6: The install file so you can look at hook_schema implementation
D7: The *.install files can help you locate more information about what changed between 6 and 8 in the update functions.
D8: The entity class

Read more

IMP-specific PHPUnit Directions

We're collecting questions from today's sprint in hopes of getting documentation together for future work.

Where do I put my tests?

Add tests to the exisiting test files.

For MigrateExecutable.php, add to:

  • core/modules/migrate/tests/Drupal/migrate/Test/MigrateExecutableTest.php

For Sql.php, add to

  • ./core/modules/migrate/tests/Drupal/migrate/Tests/MigrateSqlIdMapTest.php

What does the getMock function do? What's the high-level idea of the function and how do I know what to pass to it?

Read more

Setting up to contribute to IMP

Getting set up to contribute to IMP

  1. Visit the IMP sandbox Version control tab and follow the instructions to clone the project. The sandbox contains both Drupal 8 core and the migrate module so it takes a bit to download.
  2. Work for this project is located in core/modules/migrate. If you have access to commit directly to IMP, commit directly to this 8.x branch. Otherwise, create and submit patches to the IMP issue queue. Directions are on the Version control tab under the heading Patches. Note: chx will handle consolidating work into patches for core as necessary.
  3. It is helpful to download and install Drupal 6 in a separate directory so you can test enabling modules, configuring them, creating content and seeing how that content and configurations are stored in the D6 database.
Read more
mikeryan's picture

High-level IMP architecture

Backing up a bit to a high-level view of the architecture, right now we've got the whole thing in a single Migrate module. I'd like to see some separation between the APIs that will service all kinds of migrations, and the support specific to Drupal-to-Drupal upgrades, as well as identifying what (if anything) requires a module. The buckets of code I picture:

Core

  • Migrate API: Most of what we have now.
  • Upgrade API: Any classes specific to Drupal-to-Drupal migration.
  • Upgrade module: D6/D7 migration configurations, plus UI.

Contrib

  • migrate_plus module (or even call it "migrate", if there's no module in core?): General migration UI, D7 migrate features not going into core such as different source plugins, etc.
  • migrate_d2d module: Support for D5 and D8 sources, UI for remapping content types (i.e., support for custom Drupal-to-Drupal migrations as opposed to upgrade-all-the-things), etc.

Thoughts?

Read more
eliza411's picture

Second IMP Sprint

Start: 
2013-11-19 09:00 - 17:00 America/Los_Angeles
Organizers: 
Event type: 
Sprint

The IMP team is working hard to complete a core migration path from Drupal 6 to Drupal 8 by the end of January 2014. Join us on November 19 between 9:00am - 5:00pm Pacific for our second sprint.

The IMP initiaitve lead, chx, will be guiding developers from all over via IRC in freenode's #drupal-contribute channel.

Portland-based OpenSourcery has dedicated a team of three outstanding developers to support a face-to-face sprint in downtown Portland, Oregon. If you're in town, consider joining us! https://groups.drupal.org/node/368358

Sprint Goal

The sprint goal is complete unit test coverage for

  • lib/Drupal/migrate/MigrateExecutable.php
  • lib/Drupal/migrate/Plugin/migrate/id_map/Sql.php

It's a great chance to get involved with Drupal 8 and get up to speed with PHPUnit in core.

Current tasks

  • Among many things, chx is working on the FakeInsert, FakeUpdate, and FakeMerge classes so SQL writes in the id_map can be unit tested.
  • Others are creating a slew of migrations and integration tests for variables-to-config. Detailed instructions are available, so you can just grab an issue without much previous knowledge. Reviews are happening swiftly (less than 24 hours) and chx and others are highly available in IRC to answer question.
  • Late next week, we'll set up the sprint-day issues. If you'd like to lend a hand with that, let me know in the sprint planning issue.
Read more
chx's picture

PrepareRow gets a hook

filter_update_7003 is reading settings from a variable for each format it knows about. How do we handle this? Obviously we can cook this into prepareRow of the source class; but then how will contrib get on board? Reminder: we plan to use the entity load hook to let contrib extend the (module,delta) -> map mapping. It would make quite some sense to use the same hook as the event contribs will need to fire on but I am not sure what exactly they would change! Probably we need to provide some sort of extensibility for prepareRow apparently just not sure how. Another type of plugin??

Read more
chx's picture

Agenda for the 7/11 call

Where we are:

  1. The process syntax has been revamped completely, the new get has a test and there's documentation
  2. There's an entity handler but completely untested (as in I never ran it)
  3. There's a map process plugin but it's also quite untested
  4. bdone has written a few variables-to-config migrations (thanks!) others haven't, we are badly behind on them

Next steps:

  1. Reroll core patch
  2. In import, handle the case where $destination->import succeeds but there are no ids -- as in config
  3. Write more tests: migration and map process plugins, sql idmap. sql idmap tests should go into core patch #1
  4. Write the filter formats migration
  5. Create core patch #2 with the map process plugin, the filter formats source, the entity destination the filter formats migration and make sure all of this is tested
  6. Get people to write the variables-to-config migrations somehow.
  7. Entity destinations need the stub functionality. Not urgent.
Read more
nilesh.rohra's picture

Webinar: How to Migrate from .NET to Drupal – November 13@ 1:00 PM EST

Start: 
2013-11-13 13:00 - 14:00 America/New_York
Event type: 
Training (free or commercial)

Join Acquia and DOOR3 for an informative webinar on why migrating to Drupal CMS provides the flexibility and ROI your organization desires. With a strong practice around both .NET and open source (and Drupal), we understand all of the pros, cons and realities when it comes to CMS migrations. You'll hear an informative discussion on why this is the last move you’ll ever have to make.

Read more
chx's picture

Let's pivot

Currently the process values are a list of plugins and their configurations. This is causing a lot of headaches because of ordering. And also, noone thinks in terms in plugins. The Drupal 7 version of addFieldMapping was also taking destination as the first argument. Reading the current documentation at https://drupal.org/node/2127611 first might be helpful.

Read more
chx's picture

First core patch submitted

It's at https://drupal.org/node/2125717 . Focus is not the issue. Work continues as normal in the sandbox, next step is https://drupal.org/node/2125745: Variables to config

Read more
chx's picture

Pre-notes for the call on Oct 31th

As drupal.org will be down the document is copied to https://docs.google.com/document/d/1ubNnzQf38jp_yPKixg-x1DeiVoRvow9dLIqY... and the code is pushed to http://github.com/chx/imp

  1. Almost all of the D6 sources are done
  2. marvil07 is leading an effort to add requirements checking, in the beginning they are just modules existing and being the right schema version
  3. There's a new Source class, it's not pluggable, it wraps the source plugin.Same code as in SourceBase just separated out.
  4. The message display is in a separate class, it even has an interface, mandatory to pass to MigrateExecutable. Default implementation is just drupal_set_message().
  5. I am working on the first migration, id_map needs to in a semi decent shape.

What needs to be done:

  1. Review the current state with core in mind. Add missing doxygens.
  2. Once the first variable-to-config migration is done, we need to get another 50 of those
  3. Entity destinations. The base should be blatantly simple.
  4. url_alias destination.
Read more
chx's picture

IMP progress report #1: Past BADcamp, before core patch

So far we have established the structure of the new migrate module, there is a migration configuration entity which holds the configuration of the various plugins: source, process, destination and id map. As Views does, migrate also will have a MigrateExecutable class, responsible for "turning the gears" -- calling the proper plugins. There is a standalone, non-pluggable Row class which mostly just holds the source and destination data and does very little logic.

Read more
chx's picture

Versions and requirements

We need to version the sources for sure. I can't remember a case when a core minor release would break our queries but such things definitely happened in contrib (look at CCK it keeps the table names in a function because of this).

Read more
chx's picture

SourceBase split? Edit: yes.

When I look at SourceBase, it looks like a standalone class to me which wraps an iterator from the real source base. Should we split it off instead of having these funny named methods like performRewind, getCurrentRow? Our database classes already return an iterator, it's trivial to wrap reading a file in an iterator ( https://github.com/weppos/fileiterator/blob/master/FileIterator.class.php ) so I think this would be beneficial. Also we could skip computeCount and just use count() and make the sources implement Countable.

Read more
RedAppleConsultants's picture

Drupal Developer | Red Apple Consultants

Employment type: 
Contract
Telecommute: 
Not allowed

Position 1:

I need a developer with skills below to work on a 9-month project in Arlington, VA.
US Citizens/Green Cards only. No H1B
Must undergo a background investigation followed by a drug test, we pay for these

Job Title : Drupal Develper
Location: Arlington, VA
Duration: 9 months
• Experience with Drupal 6/7 - knowledge of all major contributed modules including Views, Panels, CCK, etc
• Custom module development experience
• Mastery of CSS / HTML / PHP
• PHP programming experience: 5 years
• Familiarity with Git
• Comfortable with LAMP Stack

Read more
chx's picture

On canonical representation

Yesterday jessehs was writing the filter formats plugin and we had a discussion live and then on IRC with marvil07 whether we should migrate the filter format table and then run a migration for each format to bring the filters over or just add the filter list to the formats in prepareRow. We are doing the latter.

Noone suggests bringing over entities and then fields separately either.

Read more
chx's picture

The UI

As far as I am concerned, the UI will contain:

  1. the database credential fields straight out of the installer
  2. Edit: A Drupal version select box it seems we can divine this from the system table.
  3. A file path textfield. (MIght need two for private files in D7)
  4. A button saying Migrate configuration
  5. A button saying Migrate content
Read more
Subscribe with RSS Syndicate content

IMP

Group organizers

Group notifications

This group offers an RSS feed. Or subscribe to these personalized, sitewide feeds:

Hot content this week