Can D7 handle enterprise-level change management features?

We encourage users to post events happening in the community to the community events group on https://www.drupal.org.
Senpai's picture

Now that Dries has concocted a business model that leverages core, I wonder if this is a good time to begin talking in earnest about making D7 into a manageable, upgradeable system with built-in change management features? Dries is one of only a few people with core commit privs in CVS, and as such, he's now perfectly positioned to listen and act on revisions to core that would make everyone's lives easier, but especially his new company's options or desires.

There's a need for small changes to Drupal that allow big business to leverage it to create and manage a system while being able to upgrade that system in a live setting. We've all seen the problems. Dries has too. Now Dries has a company and wants to begin leveraging Drupal's core to provide services to businesses, but then again, I'm sure that lots of other people out here do as well.

Start listing the problems, people! Post a quick comment on this thread outlining something that you've actually run into in real life while re-publishing changes to a D5.x site. What went wrong when you last moved a set of upgraded modules from the dev server to the live server by way of the staging server? Where were the pitfalls in your last QA process? How did you get around it? If enough people start talking about this now, maybe Drupal 7 can incorporate some brilliant ideas to alleviate a few change management issues. Maybe there's some idea out there that's brilliantly simple, and just waiting to be discovered. Start talking!

Comments

nice topic to bring up I was

tjholowaychuk's picture

nice topic to bring up I was wondering about this for higher end sites as well. I personally would love the ability to quarantine section(s) of a site which are being upgraded, or to alert certain roles, or again sections of a site that it will be down for maintenance in XXX amount of time allowing users to finish and crucial tasks etc.


vision media
350designs

Excellent question

Wim Leers's picture

That's an excellent question. Workhabit promised a new beta of AutoPilot within weeks in Barcelona. But now we're months further and still no news. (I've seen that you commented there as well, Senpai, asking for an update status.)

Many, many people are cursing at their pc because of this very painful task. It'd be really cool if an easily manageable solution would be created. And better yesterday than tomorrow.

I could be wrong

Tresler's picture

But in Barcelona bjaspan was talking about this, and I think it is where he is going with this: http://drupal.org/project/schema eventually.


Tresler Designs

Thanks, I didn't know about

Wim Leers's picture

Thanks, I didn't know about that module. I'll have to try out Drupal 6 and see for myself I guess :)

Starting smaller: an Upgrade module?

mlncn's picture

Like everyone else using a lot of Drupal, I want sophisticated change management where we have:

1) a live site with users posting content like crazy and possibly administrators making some configuration changes, and
2) a development site where we add tons of functionality with all the code and configuration this implies, and presumably some site structure and pages related to the new functionality

And we can merge both of them together. A more sophisticated version of the poorly-named Journal module (which lets admins make log messages about configuration changes) could really help with this-- changes made by certain roles could be recorded by default, with a checkbox option whether to record in any case, and can then be played back on a test site to create a combined (nearly) live content + dev configuration site that's ready to be taken live.

In the meantime, though, there's some really basic stuff that all Drupal site admins need that we could do pretty easily I think, such as an upgrade module:

  • record what modules are in use
  • disable all non-core modules for you, and maybe disable itself on the way out ;-)
  • you do your Drupal upgrade
  • the first module you enable is the Upgrade module, which will re-enable the other modules (presumably in a couple stages that obey dependencies)

Interest? Input? There are probably ways to make this smarter by tying into update status module.

benjamin, Agaric Design Collective

benjamin, agaric

Sounds good

tjholowaychuk's picture

sounds like a good first step to me, I like the idea of having the upgrade module remember which ones you had enabled. Unfortunately ( or fortunate ? ;) ) I have not had to do many live upgrades our clients seem to be content with keeping their sites as they are for the most part, but I could see this being handy.

I was very surprised I could not even find a "maintenance" module to notify users that the site will be down in X amount of time etc


vision media
350designs

Module sets

majortom's picture

I like your idea for an upgrade module that would remember which modules were in use and enable one to disable all modules quickly and then re-enable those modules selected.

Two related ideas:

1) An easy way of storing/moving to a new site a module set (i.e. a list of which modules are enabled) so that one can easily clone a working site.
2) An easy way to tell if an enabled module is in use by any content (not sure if this can be done with a reasonable amount of work). This would make it easy to eliminate modules that are no longer being used.

/carmi

Upgrade Assist

sun's picture

Talk is silver, code is gold. I have started working on a "Upgrade Assist" module that I plan to commit to Upgrade Status' project space. It tries to cover and guide the user through all steps required for upgrading a fully-fledged Drupal site, i.e. accounting for all the extra steps for CCK, CCK contrib modules, Views, Themes, aso.

If someone is interested in joining this effort, please drop me a mail.

Daniel F. Kudwien
unleashed mind

Daniel F. Kudwien
netzstrategen

Surely interesting!

Jürgen Depicker@'s picture

Autopilot

frankcarey's picture

I've been following Autopilot waiting for their solution. There was a post very recently about the code being complete..
.. here is is

http://www.workhabit.org/autopilot-beta-schedule#comment-695

This is ESSENTIAL , but I'm sure I'm preaching to the choir. I wish they would get it out there so we can start testing and see if this is gonna be the solution that We're hoping it to be.

Seems that there should be some sort of rollback system as well. User contib modules don't always behave nicely, and you should be able to beta test without giving up your site to the fate of the gods.

Sounds promising I hope it

tjholowaychuk's picture

Sounds promising I hope it works out well too. The commenting is extremely sparse which scares me a little but like you said we will have to wait and see


vision media
350designs

a proposed workflow for change management

modulist's picture

We have a few large enterprise sites out there, and we're struggling with managing the process of managing changes, especially when a site goes live.

We start with a dev server that's typically a site skeleton of lorem ipsum pages and the bare bones of a theme. This typically resides on a server inside the studio or on an extranet.

As we build the theme out and start adding content, the dev server gets upgraded to a staging server, which our client can access continually. It's protected from prying eyes and search engines by .htpasswd. If the staging server is not in the client's environment, then we create a mirror copy on the client's web server.

To go live, we simply throw a DNS switch, send out and email, and the staging server is visible to the whole world. This is where the real fun begins.

Immediately after launch, we create a new dev server at the studio. If there's a second release of the site with new features, we'll work with the site's theme and add menus, blocks, views and the like.

Meanwhile, the client needs to make content changes and post them to the live site. We can't simply sync databases during development, because the client will overwrite the changes to our blocks and menus. We can't sync at the end of the process because we'll obliterate all of the client's content changes made during development.

The simplest solution would be to have the client make content changes to a shared staging server. At the bottom of every node/*/edit page, the buttons would read as follows:

[preview] [submit] [] promote to...

When checked, the promote checkbox will add the content to the staging server and also post the content to a previously designated live server.

To enable this, we'd have set up a daisy chain of Drupal installations, each of which would have hooks to the previous and next servers in the chain. Here's an example of a full-blown chain:

dev -> staging -> production -> live -> backup -> archive

One each machine, settings.php would be modified to also include access credentials to its linked servers.

Naturally, the promote button could be restricted to certain users and roles or matched with an approve button on the /admin/content/node management screen.

@modulist

Yes yes yes!

pauldawg's picture

I don't know why there isn't more talk about this, and frankly I wonder what people are doing without this kind of feature.

I have been aggressively working on a site for the last several months and am a Drupal newbie. Once I go live (soon I hope), I will be stuck with an active Drupal Production site with user-generated content and I still don't have a plan for how to merge my ongoing development with my Production content. My desire is to release frequent iterations to improve the site, but as of now this looks quite daunting. I have seen an article in the past where someone was working on an "odd-even ID's" method of making sure that records created in Development were created with odd IDs and ones in Production were created with even, so when you do a promotion there is no conflict. Sounds interesting (though I have some doubts) but is anyone actually doing this? If so, how?

How do others manage your live Production content and still have controlled promotions of new content from your Development environment without extensive down time and manual work? Are there any tools for 6.x that will help me here? I find it hard to believe that everyone is just doing changes directly to their Production environment when a new release happens, or that they are taking the site offline while they rebase their Development environment and do changes, tests, and automated deployments. There has got to be a better way being used out there. Please help me!

Thanks so much!

Write-ups of techniques

thinkling's picture

Hey Paul,

I started to collect different techniques that people have documented a while back; take a look here:
http://delicious.com/thinkling/drupal+deployment

If you find other worthwhile links I'm happy to add them there. We could also start a wiki page to collect these in. (?)

I haven't picked a technique myself yet.

I think most people are in a situation where (almost) no content gets generated on the dev servers, so they worry primarily about moving settings forward. Greg Dunlap's Deploy module seems to be the most current and actively developed solution there, though it doesn't cover everything yet.

Great

pauldawg's picture

Thanks, this is really helpful!

Clarification

pauldawg's picture

Just to clarify, by "content" I am not just referring to nodes. Nodes are not the big issue -- there I can just copy and paste or write a selective dump (although node references and links still wouldn't work.) The biggest challenge is actually installing and configuring modules, content types, taxonomies, filters, and then deploying these changes. The dependencies are a lot more complex.

Several Approaches

gdd's picture

I have recently give a talk at a couple DrupalCamps on this issue, and the slides from my last talk are up here if you want to see a variety of approaches and their pros/cons.

http://heyrocker.com/drupal/content/data-import-deployment-slides-drupal...

There is also a video of that presentation here:

http://seadug.blip.tv/file/1033300/

Note I created the Deploy module so there may be some bias there ;) However all the approaches including Deploy have up and down sides and its worth choosing what works for you.

Greg, awesome slides, dude!

Senpai's picture

Greg, awesome slides, dude, thanks for sharing! "I can has deployment, prettyplz?" Priceless!

BTW, after our last few IRC exchanges, I really think I should introduce you to Ernest Berry, the lead dev who's working on AutoPilot. I'll send both of you an email, cause it'd be great if we could have some sort of a collaboration of the minds going forward into this problem's future, ja?
______________________________
Senpai (my d.o account)
  ~ Build a better WorkHabit ~


Joel Farris | my 'certified to rock' score
Transparatech
http://transparatech.com
619.717.2805

Sounds good to me!

gdd's picture

Sounds good to me!

-1 for odd-even ID's

emjayess's picture

That approach just sounds kludgy and inelegant... and bastardizing to the DB. To be a good solution, I think database ID's/keys should be a transparent [non-]issue to the challenge of deployment from one environment to another.

--
Matt J. Sorenson (emjayess)
d.o. | g.d.o. | WEBJAX'd! | twitter

--
matt j. sorenson, g.d.o., d.o.

UUIDs would be better

robertDouglass's picture

even/odd is also fragile. What if you're merging from three sites? UUIDs have the potential to solve the id collision problem globally.

What about automated CRUD generation?

rolly-gdo's picture

Has anyone had tried integrating MySQL CRUD generation into their development environment? It is certainly a step above simple versioning of schema/data dumps, which I think I have seen discussed in the Drupal community before. Multi-platform open-source tools exist for schema diff'ing like MySQL::Diff for Perl and Gerald for Python. For data diff'ing, mk-table-sync is available, part of the excellent Maatkit.

Imagine a development server, under SVN version control. It could just as easily be another system like Bazaar or Hg. A developer creates his new module and configures it within his development Drupal site. The database has now been changed since the last revision. He/she has CREATEd tables, DROPped tables, ALTERed rows, DELETEd rows and INSERTed rows.

The developer commits his new module to the repository. Thanks to a pre-commit script, four files are added/updated in the subsequent commit:

schema.revert.SQL
schema.apply.SQL
data.revert.SQL
data.apply.SQL

And thanks to a post-commit script, a "druplicate" is made of the current drupal database. This will provide the baseline database that MySQL::Diff and mk-table-sync will compare to, next time the pre-commit script runs.

So now you have a new revision r300:

/projectX/dev$ svn log --limit 2
------------------------------------------------------------------------
r300 | rolly | 2008-01-28 01:11:01 -0500 (Mon, 28 Jan 2008) | 2 lines

New module added and configured.
------------------------------------------------------------------------
r299 | rolly | 2008-01-28 01:02:44 -0500 (Mon, 28 Jan 2008) | 1 line

baseline

If I'm not mistaken, at this point you are theoretically able to apply the r300 code and database changes (from r299) to the staging site with something like:

/projectX/dev$ cd ../staging
/projectX/staging$ svn merge -r 299:300 ../dev .
/projectX/staging$ svn cat -r 300 ../dev/schema.apply.SQL | mysql staging
/projectX/staging$ svn cat -r 300 ../dev/data.apply.SQL | mysql staging

And also, you are theoretically able to revert r300 code and database changes with:

/projectX/staging$ svn merge -r 300:299 ../dev .
/projectX/staging$ svn cat -r 300 ../dev/schema.revert.SQL | mysql staging
/projectX/staging$ svn cat -r 300 ../dev/data.revert.SQL | mysql staging

Thoughts?

[EDIT: Sorry, this may not be the right thread to post in, since it doesn't have much to do with D7, per se.

Hmm

greg.harvey's picture

I'm not clear on why Drupal needs to change here - good development and IT management behaviour within the Drupal community is more the order of the day. As far as I'm concerned, Drupal already goes further than many other "enterprise-level" development frameworks in the tools it provides for change management. I had far more trouble as the product development manager for a massive blue-chip company (which will remain nameless, but you can look at my drupal.org profile ... ;-) managing change in Microsoft .Net web services than I have with Drupal. Proper use of the new hook_schema in D6 and .install files makes change management easy! =)

We have Production, Stage, UAT and Dev, btw... Dev is a free-for-all, which has it's database cleaned after every complete Production release (because this is a safe time to do it - HEAD and the release branch contain the same code, so it's safe to copy the database down - we have shell scripts to handle the subtle differences in site names, folder names, etc.), UAT is a semi-stable version of HEAD for people within the business to play with and comment on, Stage is sacred and only sysadmins are allowed near it - it is their sandbox once we've packaged a release and has it's database dumped from Production nightly so it always has production data from the day before - and finally Production itself.

Any database structure changes are handled in hook_update implementations, any code changes are managed by Subversion branches and any settings changes not handled by hook_update are done step-by-step, by sysadmins, following a release note. (We hope to start scripting these in Drush soon though).

+5 for good change

emjayess's picture

+5 for good change management practices/behaviors/workflows. +5 for reminding everyone that Drupal is strong & does provide a useful toolbox.

-5 for suggesting nothing more is necessary just because Drupal already goes further than many other enterprisey packages... esp. stuff from Redmond =\ The appeal of Drupal & threads like this, is that there's no limit to the brainstorm. If it can be made easier, it will be made easier. If somebody can dream it up, they can build it too (or write it down, and somebody else will pick it up). In many ways, Drupal sets its own bar for comparison!

And, not everyone who works with Drupal is working in an environment with access to a strong dev & IT/Project team, or has all those technical chops... a lot of folks are solo flyers, or collaborating with perhaps a small handful of a team for the first time, so continuous improvement in these areas is important.

--
Matt J. Sorenson (emjayess)
d.o. | g.d.o. | WEBJAX'd! | twitter

--
matt j. sorenson, g.d.o., d.o.

To clarify my position

greg.harvey's picture

-5 for suggesting nothing more is necessary just because Drupal already goes further than many other enterprisey packages...

I don't think I was clear enough ... "because Drupal already goes further than many other enterprisey packages" is not my main reason for suggesting nothing more is necessary. More that there are many, better, existing open source tools developed for this task and developing ANOTHER set of deployment tools specifically for Drupal seems like duplicating effort.

I'm all for hearing other tools people are using for Drupal deployment. I can kick things off by saying we use JMeter to write deployment scripts (test plans) to automate all the necessary UI steps between releases. So a JMeter integration module that allows you to record a JMX file from within Drupal and copy it straight out would ROCK!! (ps - I plan to blog our approach this afternoon - I promise to post. =)

I would also love to see Drush become much stronger, as there are certain things we do in JMeter now which, with a bit of work, Drush could do much more elegantly.

In fact, should Drush be considered for core? Is it being?

That's the sort of conversation I think is more valuable...?

As promised...

greg.harvey's picture

JMeter and Drupal deployment blog post:
http://www.drupaler.co.uk/blog/more-deploying-jmeter-shell-scripting-dep...

Why not Devel and macros? For a number of reasons:

  1. I don't want Devel installed on Production, taking up resources unnecessarily
  2. I don't want to have to install and uninstall Devel every release
  3. I can execute JMeter from a Linux shell script
  4. I can pass dynamic variables (such as server name/IP, user credentials, etc.) to JMeter from shell, so I can have shell scripts for each environment (dev, UAT, stage, production)
  5. I can use a JMeter for much more, e.g. basic functional tests, load testing, etc. - just by changing a few settings

I don't think it would be worthwhile trying to match these sorts of features in Drupal, or even half match them. Thinking about the JMeter integration though, the ability to export Devel module macros as JMeter JMX test plan files might be a good way to tackle things.

Another thing I'm all for doing is writing a bunch of HOWTO articles on Drupal.org specifically aimed at "solo flyers, or collaborating with perhaps a small handful of a team for the first time". Even people in these groups could save themselves endless time and effort with a few tips on good management of your code and some simple deployment ground rules. =)

Testing

sun's picture

It seems you are overlooking the fact that Drupal 7 will ship with almost complete unit tests. Contributed and custom modules are able to implement own tests and enhance the testing system.

Daniel F. Kudwien
unleashed mind

Daniel F. Kudwien
netzstrategen

Ok

greg.harvey's picture

So that checks off functional testing, to a degree, (which is a smaller concern for me, personally, as I don't write that much complex code) but doesn't deal with deployment challenges?

Ps - Upgrade Assist could prove very useful - if you liaised with the Drush guys to make it work from command line too, that would be really powerful. =)