Staging to Production without overwriting content

Posted by Mark123 on September 20, 2011 at 5:24pm

When I sync from my Staging system to the Production system, content created on the production system (such as a new node) is lost.
What are methods for using drush rsync and drush sql-sync to avoid overwriting content?
Is there some way to specify to not overwrite nodes/comments created/modified by me? (As I'm the only user on the Staging system.) Any other processes/methods?
Our site is at the point where I still need to do much more development, but that our users can begin to populate the system with content. I haven't been able to find out how to let them create content, without my new development overwriting their work. (And I need to find this out soon!!!) Thanks!

Comments

This is a complex question.

Posted by greg.1.anderson on September 20, 2011 at 5:35pm

This is a complex question. Best practices for Drupal is to only sql-sync from live to dev, and only rsync from dev to live (rsync @live:%files @dev:%files as needed).

The features module can be used to push database configuration to files, which can be rsync'ed to live and then re-imported into the database. Not all modules support features out of the box. Some people write update scripts to push their configuration changes (write an update script, then run updatedb on live).

You can also consider using git instead of rsync to move files. With drush-5.x HEAD (check out from git), you can use the scripts in examples/example.bashrc to run git @live pull to remotely pull files on your live server.

Not immediately helpful, but hope on the horizon

Posted by eporama on September 20, 2011 at 5:46pm

Also, just to remind people that we're all in the same boat, this is an important enough issue that it has its own Initiative in the Drupal 8 roadmap. You can read more about what's happening on this front at the CMI group page.

Personally, I've found Features and git to be an amazing 95% solution. There are some things that are not yet Features ready, but as much as available, it does help. However, it's also not very easy to retro-fit an existing site to be Features driven.

I have seen some success with simple sites using node_export (which has an import function as well) to export production nodes and import them into the Dev site and then locking the site to new content while you now sql-sync the databases from dev to production. Your mileage may vary and I would definitely suggest practicing this on non production versions first...

we currently use

Posted by duckx on September 20, 2011 at 7:57pm

we currently use drupal.org/project/demo and works great on managing database changes and content creation. only problem is that one person can work on the database at once. but if you want 2 ppl to work in parallel, one would need to make db structure changes and export via features, while another solely works on inputing content and then you merge them at the end via demo and features module.

but from my knowledge there is currently no way where you can do a database merge, because if you create content on dev and prod at the same time they might end up taking up the same node id.

why are you syncing the whole

Posted by dgtlmoon on September 20, 2011 at 11:00pm

why are you syncing the whole site?

some suggestions and rumors i've heard..

update your theme and deploy only via git, deploy only what has changed
switch to UUID module instead of integers for your node-id's, this I have not tried, but if it works, should do exactly what you need
use features for deployment of functionality

http://dgtlmoon.com

i want to see live content

Posted by duckx on September 21, 2011 at 10:54pm

i want to see live content and im thinking if your working on dev why not have a replica of what your going to see on prod exactly.. i mean if it was just building out the structure then yes demo wouldnt really be necessary and just features would be fine.

tell me if im on the wrong track here, as im always looking to improve my workflow.

Thanks!

Thanks for the responses!

Posted by Mark123 on September 21, 2011 at 10:20pm

Thanks for the responses! Gives me a little common sense about what the issues and options are, and some approaches to work on.
Greg, I was surprised with:

Best practices for Drupal is to only sql-sync from live to dev, and only rsync from dev to live (rsync @live:%files @dev:%files as needed).

I would have thought that some of the DB data and files would have been tightly integrated enough to need to be updated/synced together.

@mark123: Yes, it is

Posted by greg.1.anderson on September 21, 2011 at 10:58pm

@mark123: Yes, it is definitely true that in Drupal, the db and files are tightly integrated. If you use a revision of your "db" with the "wrong" set of files (newer or older code), then things break terribly. So, how can you sync only one half?

Live-to-dev is the easiest to understand. When you first deploy your site, live == dev; their files and db are identical. Then, if users add content, the db on the live site is newer. If you sql-sync live back to dev, then once again live == dev, because the files have not changed: your site will still work.

For dev-to-live, a minor update is easiest to understand. With drush, run pm-updatecode to get a new set of files, then run updatedb to update the database to match. Test thoroughly. When you are ready to take the upgrade live, rsync only the files over, and then run updatedb on the live site to make the db match your new Drupal + contirb versions. Of course you should take the live site offline for the rsync / updatedb stage.

Adding new modules and reconfiguring modules is a little more difficult (see response from @eporama, above). For simple changes, some people just do everything twice: try it out on the dev site, take your site offline, then move your files to live and reconfigure. Features gets around the need to modify the database twice by providing a means to take information that features stores in files (representing your modules' configuration settings) and make the equivalent changes in the database, so everything gets put into the state it is "supposed" to be in per the file contents (this is a 'feature-revert'). With features, then, you can reconfigure your settings on your dev machine, use 'features update' to write the db settings to files. From here, if you rsync your files over to the live site and do a 'features revert', you will merge your dev site configuration changes in with whatever new content has been added to the database on the live site, no database transfer required. Unfortunately, not all modules support features.

Many thanks for the excellent

Posted by Mark123 on September 22, 2011 at 3:51am

Many thanks for the excellent explanation!

One more question, Greg: are

Posted by Mark123 on September 22, 2011 at 2:13pm

One more question, Greg: are there any issues with live-to-dev sql-sync, as far as overwriting data that may have been developed on the dev machine in the interim?
Using this approach, are the only changes that I can make to dev in the interim changes to files? For example, I might not want to create a new content type, because some of the DB tables involved might get overwritten later with the live-to-dev sql-sync.

Yes, that's right. On

Posted by greg.1.anderson on September 22, 2011 at 2:58pm

Yes, that's right. On smaller sites, some people add content types and make other configuration changes directly on the live site. However, you can add content types on the dev site if you use features, because the features module will easily export cck types and views.

Drush sql-sync compare and update

Posted by Sree12 on February 28, 2012 at 7:18am

Does Drush sql-sync have a compare and update option.I dont want to destroy the full target database but want to compare dump from both servers and update the change.Can anyone help on this if u know.Its urgent for me.

Thanks

You can use diff to compare

Posted by greg.1.anderson on February 28, 2012 at 7:32am

You can use diff to compare the dumps if you made an ordered dump (not supported by postgres). drush_extras has an sql-hash command that can be used to compare the hash values on a table-by-table basis, to find out which tables have differences.

To compare and update i sql- sync

Posted by Sree12 on February 28, 2012 at 7:41am

Hi,

Thanks for your reply.Can u please explain what should i add more with my sql-sync command
I m giving 'drush sql-sync @dev @local --debug'.Please guide.
Thanks my friend.

Drush extra

Posted by Sree12 on February 28, 2012 at 9:10am

Hi,

Drush extra current version is 6.x.Does it matter that i m using drush 7.x

Thanks

Merge

Posted by phunster on October 11, 2011 at 3:20am

Why can't you merge the databases?

Sometimes if you are clever

Posted by greg.1.anderson on October 11, 2011 at 4:50am

Sometimes if you are clever and careful you can do a database merge. This is easy if the changes on the same site did not touch any of the same tables.

The place where it gets tricky is when you add one new node on each site. In that instance, you'll have two different nodes with the same id. If you try to renumber one of the nodes, then you need to make sure to get all of the references to it that may exist in other tables.