Can D7 handle enterprise-level change management features?
Now that Dries has concocted a business model that leverages core, I wonder if this is a good time to begin talking in earnest about making D7 into a manageable, upgradeable system with built-in change management features? Dries is one of only a few people with core commit privs in CVS, and as such, he's now perfectly positioned to listen and act on revisions to core that would make everyone's lives easier, but especially his new company's options or desires.
There's a need for small changes to Drupal that allow big business to leverage it to create and manage a system while being able to upgrade that system in a live setting. We've all seen the problems. Dries has too. Now Dries has a company and wants to begin leveraging Drupal's core to provide services to businesses, but then again, I'm sure that lots of other people out here do as well.
Start listing the problems, people! Post a quick comment on this thread outlining something that you've actually run into in real life while re-publishing changes to a D5.x site. What went wrong when you last moved a set of upgraded modules from the dev server to the live server by way of the staging server? Where were the pitfalls in your last QA process? How did you get around it? If enough people start talking about this now, maybe Drupal 7 can incorporate some brilliant ideas to alleviate a few change management issues. Maybe there's some idea out there that's brilliantly simple, and just waiting to be discovered. Start talking!
nice topic to bring up I was
nice topic to bring up I was wondering about this for higher end sites as well. I personally would love the ability to quarantine section(s) of a site which are being upgraded, or to alert certain roles, or again sections of a site that it will be down for maintenance in XXX amount of time allowing users to finish and crucial tasks etc.
vision media
350designs
Excellent question
That's an excellent question. Workhabit promised a new beta of AutoPilot within weeks in Barcelona. But now we're months further and still no news. (I've seen that you commented there as well, Senpai, asking for an update status.)
Many, many people are cursing at their pc because of this very painful task. It'd be really cool if an easily manageable solution would be created. And better yesterday than tomorrow.
I could be wrong
But in Barcelona bjaspan was talking about this, and I think it is where he is going with this: http://drupal.org/project/schema eventually.
Tresler Designs
Thanks, I didn't know about
Thanks, I didn't know about that module. I'll have to try out Drupal 6 and see for myself I guess :)
Starting smaller: an Upgrade module?
Like everyone else using a lot of Drupal, I want sophisticated change management where we have:
1) a live site with users posting content like crazy and possibly administrators making some configuration changes, and
2) a development site where we add tons of functionality with all the code and configuration this implies, and presumably some site structure and pages related to the new functionality
And we can merge both of them together. A more sophisticated version of the poorly-named Journal module (which lets admins make log messages about configuration changes) could really help with this-- changes made by certain roles could be recorded by default, with a checkbox option whether to record in any case, and can then be played back on a test site to create a combined (nearly) live content + dev configuration site that's ready to be taken live.
In the meantime, though, there's some really basic stuff that all Drupal site admins need that we could do pretty easily I think, such as an upgrade module:
Interest? Input? There are probably ways to make this smarter by tying into update status module.
benjamin, Agaric Design Collective
Sounds good
sounds like a good first step to me, I like the idea of having the upgrade module remember which ones you had enabled. Unfortunately ( or fortunate ? ;) ) I have not had to do many live upgrades our clients seem to be content with keeping their sites as they are for the most part, but I could see this being handy.
I was very surprised I could not even find a "maintenance" module to notify users that the site will be down in X amount of time etc
vision media
350designs
Module sets
I like your idea for an upgrade module that would remember which modules were in use and enable one to disable all modules quickly and then re-enable those modules selected.
Two related ideas:
1) An easy way of storing/moving to a new site a module set (i.e. a list of which modules are enabled) so that one can easily clone a working site.
2) An easy way to tell if an enabled module is in use by any content (not sure if this can be done with a reasonable amount of work). This would make it easy to eliminate modules that are no longer being used.
/carmi
Surely interesting!
Autopilot
I've been following Autopilot waiting for their solution. There was a post very recently about the code being complete..
.. here is is
http://www.workhabit.org/autopilot-beta-schedule#comment-695
This is ESSENTIAL , but I'm sure I'm preaching to the choir. I wish they would get it out there so we can start testing and see if this is gonna be the solution that We're hoping it to be.
Seems that there should be some sort of rollback system as well. User contib modules don't always behave nicely, and you should be able to beta test without giving up your site to the fate of the gods.
Sounds promising I hope it
Sounds promising I hope it works out well too. The commenting is extremely sparse which scares me a little but like you said we will have to wait and see
vision media
350designs
a proposed workflow for change management
We have a few large enterprise sites out there, and we're struggling with managing the process of managing changes, especially when a site goes live.
We start with a dev server that's typically a site skeleton of lorem ipsum pages and the bare bones of a theme. This typically resides on a server inside the studio or on an extranet.
As we build the theme out and start adding content, the dev server gets upgraded to a staging server, which our client can access continually. It's protected from prying eyes and search engines by .htpasswd. If the staging server is not in the client's environment, then we create a mirror copy on the client's web server.
To go live, we simply throw a DNS switch, send out and email, and the staging server is visible to the whole world. This is where the real fun begins.
Immediately after launch, we create a new dev server at the studio. If there's a second release of the site with new features, we'll work with the site's theme and add menus, blocks, views and the like.
Meanwhile, the client needs to make content changes and post them to the live site. We can't simply sync databases during development, because the client will overwrite the changes to our blocks and menus. We can't sync at the end of the process because we'll obliterate all of the client's content changes made during development.
The simplest solution would be to have the client make content changes to a shared staging server. At the bottom of every node/*/edit page, the buttons would read as follows:
[preview] [submit] [] promote to...
When checked, the promote checkbox will add the content to the staging server and also post the content to a previously designated live server.
To enable this, we'd have set up a daisy chain of Drupal installations, each of which would have hooks to the previous and next servers in the chain. Here's an example of a full-blown chain:
dev -> staging -> production -> live -> backup -> archive
One each machine, settings.php would be modified to also include access credentials to its linked servers.
Naturally, the promote button could be restricted to certain users and roles or matched with an approve button on the /admin/content/node management screen.
Yes yes yes!
I don't know why there isn't more talk about this, and frankly I wonder what people are doing without this kind of feature.
I have been aggressively working on a site for the last several months and am a Drupal newbie. Once I go live (soon I hope), I will be stuck with an active Drupal Production site with user-generated content and I still don't have a plan for how to merge my ongoing development with my Production content. My desire is to release frequent iterations to improve the site, but as of now this looks quite daunting. I have seen an article in the past where someone was working on an "odd-even ID's" method of making sure that records created in Development were created with odd IDs and ones in Production were created with even, so when you do a promotion there is no conflict. Sounds interesting (though I have some doubts) but is anyone actually doing this? If so, how?
How do others manage your live Production content and still have controlled promotions of new content from your Development environment without extensive down time and manual work? Are there any tools for 6.x that will help me here? I find it hard to believe that everyone is just doing changes directly to their Production environment when a new release happens, or that they are taking the site offline while they rebase their Development environment and do changes, tests, and automated deployments. There has got to be a better way being used out there. Please help me!
Thanks so much!
Write-ups of techniques
Hey Paul,
I started to collect different techniques that people have documented a while back; take a look here:
http://delicious.com/thinkling/drupal+deployment
If you find other worthwhile links I'm happy to add them there. We could also start a wiki page to collect these in. (?)
I haven't picked a technique myself yet.
I think most people are in a situation where (almost) no content gets generated on the dev servers, so they worry primarily about moving settings forward. Greg Dunlap's Deploy module seems to be the most current and actively developed solution there, though it doesn't cover everything yet.
Great
Thanks, this is really helpful!
Clarification
Just to clarify, by "content" I am not just referring to nodes. Nodes are not the big issue -- there I can just copy and paste or write a selective dump (although node references and links still wouldn't work.) The biggest challenge is actually installing and configuring modules, content types, taxonomies, filters, and then deploying these changes. The dependencies are a lot more complex.
Several Approaches
I have recently give a talk at a couple DrupalCamps on this issue, and the slides from my last talk are up here if you want to see a variety of approaches and their pros/cons.
http://heyrocker.com/drupal/content/data-import-deployment-slides-drupal...
There is also a video of that presentation here:
http://seadug.blip.tv/file/1033300/
Note I created the Deploy module so there may be some bias there ;) However all the approaches including Deploy have up and down sides and its worth choosing what works for you.
Greg, awesome slides, dude!
Greg, awesome slides, dude, thanks for sharing! "I can has deployment, prettyplz?" Priceless!
BTW, after our last few IRC exchanges, I really think I should introduce you to Ernest Berry, the lead dev who's working on AutoPilot. I'll send both of you an email, cause it'd be great if we could have some sort of a collaboration of the minds going forward into this problem's future, ja?
______________________________
Senpai (my d.o account)
~ Build a better WorkHabit ~
Sounds good to me!
Sounds good to me!
-1 for odd-even ID's
That approach just sounds kludgy and inelegant... and bastardizing to the DB. To be a good solution, I think database ID's/keys should be a transparent [non-]issue to the challenge of deployment from one environment to another.
--
Matt J. Sorenson (emjayess)
d.o. | g.d.o. | WEBJAX'd! | twitter
What about automated CRUD generation?
Has anyone had tried integrating MySQL CRUD generation into their development environment? It is certainly a step above simple versioning of schema/data dumps, which I think I have seen discussed in the Drupal community before. Multi-platform open-source tools exist for schema diff'ing like MySQL::Diff for Perl and Gerald for Python. For data diff'ing, mk-table-sync is available, part of the excellent Maatkit.
Imagine a development server, under SVN version control. It could just as easily be another system like Bazaar or Hg. A developer creates his new module and configures it within his development Drupal site. The database has now been changed since the last revision. He/she has CREATEd tables, DROPped tables, ALTERed rows, DELETEd rows and INSERTed rows.
The developer commits his new module to the repository. Thanks to a pre-commit script, four files are added/updated in the subsequent commit:
schema.revert.SQL
schema.apply.SQL
data.revert.SQL
data.apply.SQL
And thanks to a post-commit script, a "druplicate" is made of the current drupal database. This will provide the baseline database that MySQL::Diff and mk-table-sync will compare to, next time the pre-commit script runs.
So now you have a new revision r300:
/projectX/dev$ svn log --limit 2------------------------------------------------------------------------
r300 | rolly | 2008-01-28 01:11:01 -0500 (Mon, 28 Jan 2008) | 2 lines
New module added and configured.
------------------------------------------------------------------------
r299 | rolly | 2008-01-28 01:02:44 -0500 (Mon, 28 Jan 2008) | 1 line
baseline
If I'm not mistaken, at this point you are theoretically able to apply the r300 code and database changes (from r299) to the staging site with something like:
/projectX/dev$ cd ../staging/projectX/staging$ svn merge -r 299:300 ../dev .
/projectX/staging$ svn cat -r 300 ../dev/schema.apply.SQL | mysql staging
/projectX/staging$ svn cat -r 300 ../dev/data.apply.SQL | mysql staging
And also, you are theoretically able to revert r300 code and database changes with:
/projectX/staging$ svn merge -r 300:299 ../dev ./projectX/staging$ svn cat -r 300 ../dev/schema.revert.SQL | mysql staging
/projectX/staging$ svn cat -r 300 ../dev/data.revert.SQL | mysql staging
Thoughts?
[EDIT: Sorry, this may not be the right thread to post in, since it doesn't have much to do with D7, per se.