Joining up Features and Migrate for entity export?

We encourage users to post events happening in the community to the community events group on https://www.drupal.org.
joachim's picture

I've had an idea for a way of joining up Features and Migrate in order to export entities that are data rather than config. I'd be interested in what people think...

The background is that lately I'm working on a project where we have several taxonomy vocabularies that are more part of the site structure than really content, so I used the Taxonomy CSV module to export them to CSV files and wrote a quick custom module with a UI to import them back in, so our various developers could use them to get test sites up to speed.

Then lately I've been working a lot with Migrate module, importing large CSVs of Commerce Product data.

That got me thinking that I could have used Migrate to have the taxonomy CSVs imported. Which in turn got me wondering whether this could be joined up with Features.

Because after all, Features is made up of two pieces of functionality:

  • some code in a file, that a given module knows how to turn into data in the database
  • a way of taking data in the database and turning it into the code.

Migrate module knows how to turn a CSV file into data... so the missing piece is just turning the data into CSV. And a little bit of glue for Migrate module:

  • export entities to Features as hook + a CSV file, using EntityAPI metadata properties to get the data and using the property names as the CSV header
  • a dynamic Migration class that reads the CSV, automatically mapping the source fields using EntityAPI metadata
  • something that reads the hook written by Features to define Migrations and run them

I'm not sure how Features' concept of reverting a feature would work (re-import without rollback perhaps, to keep entity IDs the same?), but it could be a pretty neat way of exporting data entities to code.

Comments

This was kinda like something

yurtboy's picture

This was kinda like something I brought up in the backup_migrate module thread.
It allows you to choose the tables to backup and then via the gui or drush you can move the data around. So during the initial build out it is fairly easy to move settings, structure and or content around.
In your case you could not just pick the term but would have to update the term tables.
One other thought is the feeds module. This could also help to push data around more precisely but not structure.
I struggle with this a lot on our team as well. Features seems to often just break down at some points so we try not to use it till later in the build process.

I've had a few problems with

joachim's picture

I've had a few problems with Features, but I've been using it for a while and I've found as long as you stay clear of the more icky things to export it's fine.

Hence I use it for Views, Contexts, fields, node types, vocabularies (on D7). But not for roles, permissions, menu items, or variable settings.

Furthermore, it's good idea to avoid messy dependencies among features. I used to try to make features stack: so a node type feature, with a views feature on top, and then a contexts feature. But that let to problems. These days I split features up by site section or functionality, so a feature for the about section, a feature for each product type, etc.

Splitting a feature or moving components to another feature is a bit messy, so it's good to spend a little time figuring out a structure for different features.

Still pondering this. We

joachim's picture

Still pondering this.

We possibly don't need Features -- a custom exporter could take care of creating the CSV files.

Whether this works with Features or a custom exporter though, there needs to be a clean code-free way of getting the CSV files.

Entity API metadata seems to be the way to do this, since that's also used by Migrate for importing entities.

This basically would need to produce a CSV whose rows match the fields that Migrate expects to find for its sources.

I've made a sandbox module

joachim's picture

I've made a sandbox module with some of what I have in mind: http://drupal.org/sandbox/joachim/1692846

What about attached data files?

bailey86's picture

Hi,

Just a quick note - if you are looking to export/import modules and their attached files you may be interested in:

http://drupal.org/sandbox/bailey86/1278830

bailey86's picture

If you check out http://drupal.org/sandbox/bailey86/1278830 you'll see that I've worked on this already,

I've just completed the D7 port.