I have a project that involves a nightly importing of data from an external source, and from what I've seen Migrate module would be a more robust solution that me writing code. I've been looking at it but haven't been able to figure out how to implement it to fit my needs, so I'm hoping someone here might be able to shed some light on it for me.
The tool I'm building is a middleware tool for a bike shop to import inventory data from a provider, add some data to it, and then export it out to a CSV file. I already have a drush command to import the data from the external source (a SQL Server db) into MySQL. The issue I'm trying to get worked out is getting the data from my custom table into Drupal nodes. I wrote some custom code to do it, but it seems like using Migrate would be a lot cleaner and more robust.
Every night when I update my local copy of the data, I need to determine three sets of data in comparison to the previous import:
1) what's new
2) what is no longer there
3) what has changed
This also includes getting an image from the external server based on a path in the imported data. Once I have the records identified, I need to import them into Drupal nodes. There are two ways I track the status of each node:
1) a CCK field called Status that has three possible values: new, change, delete
2) A log table that records each status change.
So if a record is:
new - I need to create a new node with a status of New
deleted - edit existing node, change Status to Delete
changed - edit existing node, change Status to Change
and in all three cases, update the log table (I then display that log data in each node).
I can write hooks to do the Status field and log updates. The biggest issue I have is getting the data into nodes, either creating or updating, and doing it in an automated fashion. From what I've seen, Migrate should be able to handle it fine, but it looks like a pretty steep learning curve. If I'm seeing correctly, it's all through code and not a UI, and there doesn't seem to be much (at least up to date) good documentation (migrate.api.php is empty). Any light that could be shed on this for me would be greatly appreciated.
Thanks.
Comments
Definitely look into using
Definitely look into using Feeds! Migrate is more for one-time complex migrations, as far as I know :)
Not sure how complex the sql db is, but might be best to queryout it into a csv on the filesystem using cron, then have feeds use the file fetcher, which can be set to either accept a file or read from a static file on the filesystem (which is what you want). You can use a node processor as you please to generate the nodes, and if there's a unique id in the csv, then you can even auto-update and auto-delete nodes using built-in feeds features. A module called feeds_tamper might also help with simple transformations, or you could hard-code the transformation in a custom module, using one of the hooks (I forget the exact name, but I receives an object with the complete results as an argument, and you can manipulate it, if I remember correctly).
Anyhow, those are my thoughts. I'm sure there are plenty of ways to do this. :)
Migrate does the job
Migrate is more for one-time complex migrations, as far as I know :)
Actually, that is incorrect. Since I first posted this, I've discovered that Migrate does exactly what I need. Migrate module keeps track of the rows you have imported from a data set and what nodes they created, so when you run the import from the same data source, if you don't specify --update, it only adds new nodes, but if you do specify it, it updates existing nodes. In addition, there are pre and post migrate hooks where you can do anything you want with the data on a per-record basis. I use them for custom data manipulation and logging. Migrate has drush commands, so I just run the migrations nightly from a set of cron jobs, and everything is completely automated.
Hi Wonder95, would you want
Hi Wonder95,
would you want to share how you are using the migrate (I assuming the migrate_backup module) module to import content to a drupal site. I have site that also contains a customized blog, and I'd like to import the content from the current site to another one. I know I can push the entire database to another server, but I'd only like to import the content.
I would assume that the second caveat is file migration, but that can be attacked later.
Thanks,
Sebastian
Using Migrate, not migrate_backup
Actually, I'm just using Migrate v1 (not migrate_backup). In my case, as I detailed above, I pull from an external SQL Server database (details here on my blog) into my Drupal db using a custom drush command that's run via cron. That is all done outside of Migrate. Then, I just created a mapping in the Migrate UI that tells each db field where to go in each node, and then call drush migrate-import with the --update parameter. This adds new nodes and updates existing ones. I use the pre and post migrate hooks for some custom manipulation (like logging), and that's it.
<?php echo "hello";?>
<?phpecho "hello";
?>
How do you handle deletions
You mention passing a status of "delete" in your migration. How are you actually managing the deletion of the Drupal content in these cases?
Chris Desautels
"You mention passing a status
"You mention passing a status of "delete" in your migration. How are you actually managing the deletion of the Drupal content in these cases?"
Make that two of us that want to know the answer.
I don't recall deleting nodes via Migrate
Hmmm, I went back and looked, and I don't see where I mentioned that. I did say in my initial writeup that I needed to mark nodes in Drupal as deleted if they were no longer in the source data I was importing (so I could pass that on to the ecommerce platform, Magento), but I don't recall deleting Drupal nodes.
This was actually using Migrate v1, and was a while ago, so I don't remember all the details, and I've been using Migrate v2 since then.
Sorry I can't be more help.