Posted by mikeryan on March 27, 2009 at 12:37am
The Migrate module is already garnering some good feedback. My immediate plan is to stablize the current code base of V1 - please discuss ideas for further extension and refactoring here.
Thanks.
Comments
What more can you see happening in v1?
It seems like some house cleaning could still go on without changing anything about how the module functions, and maybe that would be acceptable to include in v1? I think splitting out all the user stuff from migrate_destinations.inc to a user.inc would be a start, then putting some of the more custom code into their own hooks or a separate submodule (the content_profile stuff comes to mind), thus cleaning up the migrate_migrate_destination_import_user() function. To move it out, you'd have to have a "complete" hook for user, like you already have for node, but that seems like it is easy and should make for a more consistent hook system. Thoughts?
Frank Carey
TwelveGrove Drupal Development
http://www.twelvegrove.com
Frank Carey
Twelve Grove Inc. Drupal Development
Yes, I'd like to do some
Yes, I'd like to do some basic file reorganization for V1 - that does qualify as house cleaning in my book. That's something I'd like to get done Sunday (my next good chunk of time for contrib work).
I'm about to start work on user migration for our current paying client, so a lot of cleanup of this area should come out of that (including getting hardcoded dependencies on content_profile and email_registration out of there).
In terms of revisiting the hook structure, adding new hooks etc., that's something we'll need to live with for a while so I don't want to do that off-the-cuff, I want to take the time to carefully review that area (with plenty of input) before committing to changes. So, I'm thinking in terms of committing a V1 release (at least beta) as soon as all critical issues are fixed then making hook refactoring the main theme of V2.
Thanks!
Mike Ryan
http://mikeryan.name/
Mike Ryan
Oh, but the user completion
Oh, but the user completion hook definitely looks like a V1 feature, if only for symmetry with nodes (and the fact that I'll probably need it on my current project:-).
Mike Ryan
http://mikeryan.name/
Mike Ryan
great!
great! let me know if you need help. I posted an issue for the house cleaning stuff.
Frank Carey
TwelveGrove Drupal Development
http://www.twelvegrove.com
Frank Carey
Twelve Grove Inc. Drupal Development
So... my thinking about V2. Drupal to Drupal
Some thinking I had when I saw the presentation at drupalcon is drupal to drupal migrations. I see two use cases.
Old drupal to new drupal
*This is my use case right now, 4.7 to 6.10. *
There are a lot of old sites out there that need to go from 4.7 or 5 to 6+. The traditional upgrade path of small incremental steps. 4.7 -> 5 -> 6 -> 7 works if you have a rather default install, but a lot of modules are no longer supported for new versions, especially old custom ones. The upgrade process can involve many steps, all prone to error, and a lot of wasted time. I've found migrate a great tool to just import all the data from the old site into the new one. It is allowing me to expose the data to my code, and massage it as i need to reformat or delete garbage data that has built up. I also take advantage of all the current validations, not just in core, but also contrib.
On the other end, you often have new modules (cck, user_relationships) that weren't even available before, that you might like to create content with based on previous content or as a replacement for it.
The benefits to this approch:
+ I can design a new site based on current best practices and functionality, not based on old, probably not-so-best practices.
+ I can create my new site BEFORE migrating any data, and not have to be too worried about upgrade paths.
+ I can QUICKLY TEST with limited numbers of "test" migrations along the way to see actual data but not having to use import everything for every test.
+ I can make new data and ignore or change old. I may not want to bring over every node type in the old system. Also I may want to create brand new content (content_profile).
+ In theory, you could migrate as you go. Instead of doing 1 giant import, you could keep looking for new/updated content and users only. You could also have both site in-operation at the same time. Dev team can make nodes and users on the dev system without having to worry about unsyncing the databases.
What i would like to see is more development along these lines, perhaps as another module. One that would make many common pieces of this easier. Maybe have all necessary tables be automatically included, made into views, and processed. Anything not stored in the users table, or has multiple values (roles) are a little tricky to do right now.
switching modules
I think migrate could be a great api for modules to more easily provide upgrade paths, and "crossgrade" paths to alternative modules. Wanna know what the upgrade path from profile or content_profile is to fields in core? who cares! Giving modules this type of api to import and export content from module to module would be a KILLER feature. No more being stuck with crappy unsupported modules. Should be a simple gui interface to just map the fields between the two and the Process.
There is my grand vision :) Feedback welcome.
Frank Carey
TwelveGrove Drupal Development
http://www.twelvegrove.com
Frank Carey
Twelve Grove Inc. Drupal Development
API & export
I've been working at exposing configuration for the Table Wizard and Migrate modules through APIs - migrate_save_content_set() etc. This has benefits for the project I'm on right now (doing the configuration through update hooks lets other developers replicate the environment without manually setting things up; and it puts configuration changes under revision control). But I also have in mind the ability to export configurations, much like Views export now, as PHP code to establish all the necessary tables/content sets/etc. Thus, if you've developed a configuration for D4.7->D6.10 migration, you could export and share it.
Definitely a V2 feature...
Mike Ryan
http://mikeryan.name/
Mike Ryan
writing systemx2drupal projects
My migration experience is based on two projects - the phorum2drupal module and a similar customsystem2drupal project. I was going to ask if there is some way to write a configuration file or code for Migration that would allow us to package things like wordpress2drupal and systemx2drupal into a single, downloadable set of files.
It seems like this ability you mention of exporting configurations would handle that, right? Thanks!
Assuming so, the next question is where to store these configurations - like a "Migrate bonus pack" module or something.
--
Growing Venture Solutions | Drupal Dashboard | Learn more about Drupal - buy a Drupal Book
knaddison blog | Morris Animal Foundation
That's one idea...
...Although, since any significant migration project will involve implementing custom hooks (thus, building a module), I'm now thinking the best approach may be contributing custom modules - wordpress2migrate etc. By building an API in migrate for all configuration aspects, a custom module can do all the necessary content set creation etc. in an install hook. That's actually what we're doing on our current project - if the CMS we were migrating from were a common one, we could simply contribute the custom migrate-based module to drupal.org. And then the community would participate in refining it...
Mike Ryan
http://mikeryan.name/
Mike Ryan
I have some migrate integration moduels done.
Hey greggles, I have some migrate integration moduels done.:
migrate_message (privatemsg)
migrate_relationships(user_relationships)
migrate_userpoints (user_points)
Each one uses the module's api (if present) to import the data, but some hackish type things were sometimes needed to get everything working. It would be hard for me to imagine being able to "export" such functionality, unless you can export the php and additional functions as well. This would probably be easier if you just want core modules, but each contrib module needs its own case handling. With a good contib api, the modules are small and quick to code / reproduce once you get the migrate api down.
I'm thinking about how to contribute these back. Individual modules vs. one main project with sub modules.
The migrate API could probably use a reevaluation as well based on what where migrate helps you and where it gets in your way. Maybe we could get on irc one of these days and discuss? Mike?
Frank Carey
TwelveGrove Drupal Development
http://www.twelvegrove.com
Frank Carey
Twelve Grove Inc. Drupal Development
Export mappings
It seems like we could certainly import/export the field mappings easily enough.
Frank Carey
TwelveGrove Drupal Development
http://www.twelvegrove.com
Frank Carey
Twelve Grove Inc. Drupal Development
markdown not working??
Frank Carey
TwelveGrove Drupal Development
http://www.twelvegrove.com
Frank Carey
Twelve Grove Inc. Drupal Development
Non-view sources.
I'm having to do a lot of handcoding to bring in my old profile data and turn it into a new content_profile on the new site each time a users is imported. One of the things I'm missing terribly with so many fields is the ability to map non-view fields to content_profile fields. Everything still works, but I have to handcode the mappings :(. It would be great if we could abstract the "source" level in the stack so that anything could be a source, and I could use the current interface for those mappings as well. Probably v2?
Frank Carey
TwelveGrove Drupal Development
http://www.twelvegrove.com
Frank Carey
Twelve Grove Inc. Drupal Development
Non-view fields?
What's the nature of these non-view fields? What prevents them from being incorporated into a view?
Thanks.
Mike Ryan
http://mikeryan.name/
Mike Ryan
Multiple value fields
A good example of where we hit a wall is that we are only using left joins to join tables. This means if you wanted to import attached files for a node, you have to custom code that to happen before or after the node is saved, writing your own sql by hand.
It would be more flexible if we could write our own sql for some of the parts views can't handle and allow that query to be a source, or to define additional sources other than views.
Frank Carey
TwelveGrove Drupal Development
http://www.twelvegrove.com
Frank Carey
Twelve Grove Inc. Drupal Development