Merging databases from multiple developers

We encourage users to post events happening in the community to the community events group on https://www.drupal.org.
rootdownmedia's picture
Posted by rootdownmedia on January 8, 2010 at 4:02am

Hey all,
I've been scouring the web & forums for a foolproof method to merge databases from multiple developers. If I have developer 'A' working on theming certain parts of a site and developer 'B' working on theming others, I need a way to keep all changes in sync.

I have an SVN repository set up to handle web files. This works great for files but I cannot find a similar system for keeping my databases in sync.

I know there are several modules out there to assist with this (DatabaseScripts, Deploy etc) but none of these seem to offer a full, comprehensive solution.

How do others deal with this problem?

If need be, willing to pay a small consulting fee to help develop a solution.

Thanks,
Arthur

Categories: , , ,

Comments

Typically we just don't

Posted by bleen on January 8, 2010 at 2:45pm
bleen's picture

Typically we just don't ...
Our workflow generally follows this path. (We have a local dev environments (MAMP), a shared dev environment, & production)

  • Develothemer checks out code from SVN to a local dev environment
  • Develothemer downloads copy of DB from production environment using backup&migrate module (backup on prod, restore on local dev)
  • Develothemer does some develotheming and checks in code to SVN. QA happens on shared dev by (a) backing up DB on prod and restoring DB on shared dev; (b) doing an SVN up on shared dev; (c) and manually making DB changes on shared dev (I'll come back to this in a sec) and if it passes QA, follow the exact same process (b) & (c) on production. Possible DB changes include:
    • Creating new CCK types, or views or panels - for major changes like this, there are import/export tools available which we use to migrate up (for example) a new view
    • Config changes for simpler modules (or modules w/o import/export) - this is the most annoying; frankly we just go through the same forms and check off the same boxes and fill in the same fileds for this.
    • install new modules/themes - this is usually very straightforward to reproduce.

    That should be most possible change. Note: With this workflow, content creation should be done directly on the production env (or if you have a staging env that can be used too, but that gets more complicated.)

  • The next day the develothemers start the process over by doing an svn up and backup/restore of DB

The key to this workflow is that you always backup/restore in the Prod->dev direction; NEVER the other way around.

Views via SVN

Posted by gnat on January 9, 2010 at 1:37am
gnat's picture

Since Views allows other modules to define their own custom views, we get the ability to move views around as files for free. The first step is to create a custom module, or add to an existing one. Then use the bulk export tool (admin/build/views/tools/export). This will output code that you can copy and paste into your module.

The workflow ends up looking like this: Developer makes changes to the view with Views UI module. Developer then exports that view, and (re)places that view in the custom module. The developer then commits the changed module to the svn repository. Once the other developers update their working copies, they will need to clear their views cache, but the changes to the view will be there. With this method you should even be able to turn off Views UI on the production site.

I should also add that the above workflow is the one I have been using on almost every project lately; it actually seems to be are really popular method. If you go that route, the one thing that we have found to work quite well is to keep a narrative document about your database changes. With your list in hand, when it comes time to push those changes live by clicking through forms, nothing gets forgotten.

just so you know, this is an

Posted by arh1 on January 9, 2010 at 12:11am
arh1's picture

just so you know, this is an age-old issue that many folks have spent a lot of time trying to solve! the basic problem is that so many configuration options are set via the web UI and saved in the db.

we use a nasty manual method not unlike bleen18 above, but are trying to move to a better solution that can be properly tracked by revision control. in addition to the deploy module, have a look at features, and custom approaches like this one from Sacha Chua. i'd love to see the community hone in on a canonical solution to this problem, though.

good luck!

Thanks all for the quick

Posted by rootdownmedia on January 9, 2010 at 4:59am
rootdownmedia's picture

Thanks all for the quick responses.

It seems like Features and Context modules are a good solution. Bleen18 have you ever looked into those?

Another approach I am thinking about is using a shared database and using SVN to manage all codefiles.

This is all quite frustrating... appreciate the help.

--
ROOT DOWN!
Design & dev specializing in music industry
Brooklyn, NY

A tale of frustration

Posted by bleen on January 9, 2010 at 5:04pm
bleen's picture

I've actually had "Investigate Features Module" on my to do list for longer than I care to admit...

As for your "shared db" approach... I suspect you're not going to like that solution. Here is a sample use case that illustrates why:

  1. 9:00 am: develothemer is told by clientfromhell "change view X (display alpha) to be cooler and more spiffy"
  2. 9:05 am: contentomaker is done with first cup of coffee and starts making content
  3. 10:00 am: develothemer has done a great job rejiggering view X on his local dev box and he is now fixing all the .tpls and css becuase the changes to view X have made the site look soooo wonky that you can't even see your navigation anymore - theme changes will take an estimated 3hours
  4. 10:00 am - 1:00 pm: contentomaker has been sitting on his thumbs, frustrated, because he cannot do any work at all. SOMEONE has changed view X and now contentomaker cant navigate anywhere. He resigns himself to waiting until SOMEONE fixes the theme so he can resume work.
  5. 1:00 pm: develothemer has successfully carried out her task and shows it (still on her dev box) to clientfromhell. Clientfromhell responds "thats not at all what I wanted... put it back the way it was.
  6. 1:05 pm: It was simple for develothemer to revert her working copy using SVN revert, but now she has to fix view X. She doesnt quite remember what all of the settings used to be but she is on top of her game and was smart enough to export her view to a random txt file on her desktop before she made changes. This shouldnt be too hard to revert the view...
  7. 1:10 pm: develothemer's coworker themeodeveloper overhears develothemer complaining about her wasted time and notices that she is about to revert viewX ... AHHHH but wait!! themodeveloper has been making changes to view X (display Z)
  8. 1:30 pm: develothemer & themodeveloper powwow and come up with a plan it will take another hour to sort everything out
  9. 2:30 pm: view X (display Y) has been restored to its former glory and view X (display Z) is up and running. contentomaker has done no work from 10:00am - 2:30pm

Ha! Hysterical. But yea I see

Posted by rootdownmedia on January 9, 2010 at 5:11pm
rootdownmedia's picture

Ha! Hysterical. But yea I see your point.

It seems like your approach (listed earlier in thread) applies to sites that have already launched. Does it also apply to sites in early development (sandboxes, repo and staging server)? I ask b/c I will be adding a lot of content directly onto my sandbox, so I have some nodes to work with while Im theming.

Really appreciate everyones input. Ill do my best to contribute whatever I can!

--
ROOT DOWN!
Design & dev specializing in music industry
Brooklyn, NY

Our workflow works (for us)

Posted by bleen on January 9, 2010 at 5:15pm
bleen's picture

Our workflow works (for us) in both situations ... early dev and post launch...

1) for early dev, we just use the super-cool "devel generate" module (part of devel) to make bogus content for theming, etc...
2) for post launch, we are grabbing the whole DB (content included) form backup & migrate so we already have content to work with

Genius. Just checked out the

Posted by rootdownmedia on January 9, 2010 at 6:28pm
rootdownmedia's picture

Genius. Just checked out the SVN blog post on your site.

Are you available at all on phone or in person (as a consultant) if I have more questions?

--
ROOT DOWN!
Design & dev specializing in music industry
Brooklyn, NY

I've sent you an email so we

Posted by bleen on January 9, 2010 at 6:54pm
bleen's picture

I've sent you an email so we can take this offline...er...online...um... so we can speak privately.

What about drush?

Posted by awolfey on January 11, 2010 at 11:04pm
awolfey's picture

Bleen,

Any way to make this work with Drush on both local and live?

$dev = ($_SERVER['HTTP_HOST'] == 'local.bleen.net');
$db_url = $dev ? 'mysqli://root@localhost/db_name' : 'mysqli://username:pAsSwOrD@mysqlhost/db_name';
$db_prefix = '';

$_SERVER['HTTP_HOST'] doesn't seem to apply, therefor defaults to the live db.

Thanks

yes ... for Drush support I

Posted by bleen on January 12, 2010 at 3:37am
bleen's picture

yes ... for Drush support I typically use some unique value in the $_ENV superglobal ... something like this:

if(isset($_SERVER['HTTP_HOST'])){
   $server = $_SERVER['HTTP_HOST'] == 'local.bleen.net' ? 'dev' : 'prod';
}else{
   $server = $_ENV['user'] == 'local_user' ? 'dev' : 'prod';
}

$db_url = $server == 'dev' ? 'mysqli://root@localhost/db_name' : 'mysqli://username:pAsSwOrD@mysqlhost/db_name';

At DrupalCamp 7 Dan from

Posted by orbgasm on January 9, 2010 at 4:48pm
orbgasm's picture

At DrupalCamp 7 Dan from Agaric actually presented a method using Git with Capistrano that achieves what you are looking for (merging multilpe developers' work, and I think this method retains timestamped changelogs per user submission so that indvidual changes can be reviewed).

Might be something to look into, though I'm not 100% on what advantages it may have over standard SVN and what its drawbacks might be.

I'm personally reading up on Features, it's pretty dope.

Yes, everything-in-code plus features is a good approach

Posted by mlncn on March 20, 2010 at 9:17pm
mlncn's picture

Some raw notes / links here:

http://data.agaric.com/node/480

Working on releasing a write-up of updates in code with features assist best practices.

benjamin, agaric

Check out this link from

Posted by rootdownmedia on January 9, 2010 at 5:05pm
rootdownmedia's picture

Check out this link from Development Seed.

Seems like a great way for adding functionality (views, content types, blocks etc) but it doesnt cover migrating actual content (nodes).

Anyone have a simple way to deploy from an SVN repo to a staging server?

--
ROOT DOWN!
Design & dev specializing in music industry
Brooklyn, NY

Experience with Context & Features?

Posted by hortitude on January 13, 2010 at 8:34pm
hortitude's picture

Does anyone have any first hand experience with context & features working?

From reading the description it sounds like it would help quite a bit, but just wondering whether it works as promised?

another discussion about this

Posted by bleen on January 13, 2010 at 2:29pm
bleen's picture

another discussion about this has sprung up at http://groups.drupal.org/node/44232

So exporting / importing

Posted by rootdownmedia on February 10, 2010 at 3:18am
rootdownmedia's picture

So exporting / importing module config settings, views, content types etc are straightforward using the Features module.

How does everyone solve the content migration problem? There are a few modules out there (biggest ones are DBscripts & Deploy) but neither seem to be foolproof.

Anyone have a solid dev > prod workflow?

--
ROOT DOWN!
Design & dev specializing in music industry
Brooklyn, NY

New York City

Group notifications

This group offers an RSS feed. Or subscribe to these personalized, sitewide feeds:

Hot content this week

See all hot content.