Data Export Import module - testing and reviewing would be appreciated

bailey86's picture

Hi,

NB - The module has now been extensively developed and both Drupal 6 and Drupal 7 versions are available. The module is at:

http://drupal.org/project/data_export_import

This module is designed to be part of a whole dev/staging/live deployment process.

Basically, you can push development changes up to a beta copy of a new site - and then using the Data Export Import module you can export data from a live site and then import that data into the beta site to produce a new version of the site.

BTW - The best way to test is to install on a Drupal site - export data to a file - then alter the existing data - then re-import the data file. This will reset the data back to what it was when it was exported. There are checks in place to make sure that when importing the data the vocabularies and content types match etc - this is to prevent data being imported into Drupal site which don't have the correctly matching vocabularies/content types etc.

Later I may be able to add a profile to export/import vocabularies which can be used before exporting/importing the vocabulary terms.

Thanks for any help with testing and reviewing.

AttachmentSize
export1.png87.58 KB
import1.png98.37 KB
new_export.png99.49 KB

Comments

Existing modules

bailey86's picture

BTW - here is my summary of the existing modules:

WHY DATA_EXPORT_IMPORT IS DIFFERENT FROM EXISTING IMPORT/EXPORT MODULES

Used as part of a new type of deployment plan

The focus of this module is to provide a tool to help with the way Drupal sites are developed, tested and deployed.

Most deployment plans push updates from dev to testing to live. This means that it is not possible to safely carry out large changes. If a site has been extensively redeveloped then there is a risk that when the changes are pushed on to live the site will break. Of course, it's possible to test on a copy of the live site as much as possible and it is possible to have a secondary copy of the live site to fall back on if the new live site breaks.

A better approach is to separately develop the new copy of the site as a beta copy - changes made by developers can be pushed up to a central beta copy of the site. This site can then be extensively developed and can be completely different from the existing live site. Using the Data Export Import module it will then be possible to export live data (new users, nodes, taxonomy terms etc) from the existing live site and import that data into the beta site.

When the beta is ready to be deployed a final export of the live site data can then be imported to the beta site - and the beta site set to be the new live site. The advantages are:

  • The beta site can be extensively tested with a full copy of the live data before going live.
  • The existing live site is untouched and can be re-enabled if the beta site has any serious issues.

Designed to be easily extensible

Because Drupal can be extended and customised in any way then this module needs to be extensible. Existing delpoyment modules depend on module developers to extend their module to fit in with the deployment module.

The Data Export Import module is designed to be easily extensible.

For example, it may be the case that a Drupal site has been heavily customised and modified and contains very specific types of data. This could be exotic content types of it could mean data which is specific to a custom module.

Using Data Export Import it will be possible for the site developers to add on new profiles which are specifically designed to export and import that site's data. For exmaple, it will be possible to add on a profile called 'Buggins Widgets delivery schedules profile' which will export/import data specific for that complany's website.

There will be some common profiles which will be useful for standard sites; taxonomy terms, users, basic nodes etc. It is planned that there will be a system of sharing profiles which may be of use to others - either used directly or to be built on and extended.

No existing module is suitable

After very extensive research into the existing import/export modules it was found that none of them were able to export and import data in a suitable way.

  • Deploy module.

The big underlying assumption with the deploy module is:

'Deploy 6.x-1.x assumes that you have two exact clones of you code and configuration across your staging site and your production site (expect the Deploy and Services specific stuff). You can't have anything that differs the sites from each other. Nothing.'

The Data Export Import module is designed to allow for data to be exported to dataset files which can then be imported into an different (e.g. beta) version of a site.

Also, Deploy transmits data directly via the services module. The Data Export Import module exports to dataset files which can be transferred in different ways - and data can be re-imported back in to the exporting instance which is useful for testing purposes. I.e. Developers can work on a site and make drastic changes to taxonomy terms and nodes etc and when testing is finished the original data can be re-imported from a dataset file.

  • Features module

The Features module is focussed on exporting/importing configuration settings. The Data Export Import module is focussed on exporting and importing content type of data such as nodes, taxonomy terms, users etc.

  • Stager

Focussed on pushing from staging to live. Also it is Drupal 7.x only.

  • Backup and Migrate

The Backup and Migrate module works at the database table level. Since there are so many interdependencies between tables it is not possible to cleanly extract datasets in a way which means they can be re-imported. The Data Export Import module works at the API level to recreate datasets exactly.

  • Staging

From the staging project page:

'This module do not work very well on sites that contains user generated content, on live site. So if you need this modules's name or url or what ever, let me know.'

  • Migrate

The primary focus of the Migrate module is the importing of data from non-Drupal sources. Also, it depends on several modules including, Dbtng, elements, autoload etc. It also relies on Drush and although Drush is an excellent tool it is not used by all developers.

The Data Export Import module is designed to export/import Drupal data and is designed to be a standalone module which just uses the Drupal API to carry out it's functions and should not need to rely on other modules. Also, the user interface will be set up as cleanly as possible to enable relatively inexperienced Drupal developers to export and import datasets. Although it is planned to be able to use Data Export Import module via Drush this will not be a requirement.

The Migrate module also requires that module developers add extensions to their modules. The Data Export Import module is designed to be extensible itself - so developers can add custom profiles to match their exact site requirements.

Deployment plan summary

bailey86's picture

And here from the submission doc is the description including the deployment plan:

OVERVIEW

The purpose of this module is to provide a way to move data between Drupal instances.

This is not easy to achieve as Drupal stores configuration and data in the same database. Also, Drupal can be extended and customised which makes creating a standard data export/import module effectively impossible.

This module will provide a framework for exporting datasets as files and also for importing these dataset files. That way if the module is installed on two separate Drupal instances it will be possible to export data as a dataset file from one instance and import it into a different Drupal instance.

This module will be customisable and extendable by adding new profiles to take into account different Drupal builds. This means that if a Drupal instance has a very specialised setup it will be possible to create a new dataset profile to export/import data.

The module should include some initial dataset profiles which can be used on standard Drupal instances. These would enable the following datasets to be exported/imported.

  • Taxonomy vocabulary terms
  • Basic nodes such as page, story
  • Users

When exporting all data should be exported - when importing the data should be used in a way to make sure that the receiving Drupal instance ends up with matching data. Careful checks will be needed to ensure ID's match, that the data remains consistent.

Effectively this module will be similar to the backup_migrate module except that the backup_migrate profiles will be extended. Currently the backup_migrate profiles only allow for the choice of which tables will be exported/imported and a few other settings. This module will use the Druapl API to read data and produce rich data files which will contain all data needed to reproduce a particular dataset. It will also use the Drupal API to recreate data. It may also be necessary to use SQL directly to export/import data.

The primary usage for this module is envisioned to be exporting datasets from a live Drupal site and importing this live data into a new updated version of the site which developers have been working on - i.e. a beta version. Once the import has completed then the newly developed site can become the new live site. This has the advantage of being able to fall back to the original live site if there is an unforseen problem between the new version of the site and the live data.

Very importantly, this module will also be extensible with additional profiles being able to export/import data of any format.

Because Drupal can be extended and customised in any way it may be that certain sites contain exotic content types or even very specific custom data. Users will be able to create profiles which will be able to export/import for their specific site in conjunction with this module.

It is further hoped that it will be possible to set up a way for module users to be able to share profiles so they can be re-used or extended as required.

Before importing data it will be necessary to carry out proper checks. E.g.. when node data is being imported it will be necessary to check that the receiving site has the correct content type and fields etc. It may be necessary to use the UUID module to provide unique ID's.

All content types added

bailey86's picture

This module has a dev branch called all_content_types where all different content types can be exported and imported - keeping their original ID's.

I'm currently adding the ability to handle uploaded files and the CCK fields FileField and ImageField.

Thought if anyone is interested they would want to look.

Also, if anyone could review the project and sett it to RTBC I'd appreciate it. Project issue is at:

http://drupal.org/node/1330454

All content types now merged in

bailey86's picture

This module will now list all a site's content types for exporting. These dataset files can then be (re-)imported.

Could any bods please test this module as much as possible? Any feedback gratefully accepted. In fact, helping out via the issue queue would be a great help. Issue is at:

http://drupal.org/node/1330454

There are some notes about file sizes etc at the bottom of the issue queue.

Content migration, import, and export

Group organizers

Group notifications

This group offers an RSS feed. Or subscribe to these personalized, sitewide feeds:

Hot content this week