Posted by advseb on January 5, 2009 at 1:27pm
Hi folks,
it seems this group stalled a little bit, so lets try to give it some activity :-) I was wondering if this group has already collected some best practices around Drupal deployment and change management. So far, I have only found best practices for managing the source code of a drupal site, but I have not found anything how to transport configuration changes (done in the DB) from a devel environment to a production environment. As far as I can see there are no stable modules or scripts for that, at least not for D6. Any ideas where to look?
Regards,
Sebastian
Comments
I'm working on the Patterns module these days
I'm devoting a fair bit of time to the Patterns module these days, which is basically a really cool way to replicate any administrative or configuration changes made on one server to another server(s). Patterns is built and maintained by Gravitek Labs, and even though it's in alpha, it works surprisingly well!
There's a bunch of documentation still to be written about it, but the basic jist is this. You write a series of XML 'instructions' for how you want things to be on your site, and then Patterns can run this pattern against a live, functioning site and make the changes, rolling things back to the previous state if something should go wrong. Patterns can be nested, so you might want one pattern to call six others while installing an entire site, or maybe this time you only want to run the pattern that creates your content types or your hierarchical taxo terms for you. There's a second companion module in the works that will abstract this configuration API into a set of public functions that other modules can access during their installation process. Thus, a module could ship with a set or sets of patterns, enabling it to be installed in multiple ways on an already existing site, and without having to rely on running an install profile (again).
This whole thing has been a long time coming, and I've been preaching the virtues of smart modules and configuration APIs for a couple of years now, so it's nice to see that someone's putting a VERY hefty bit of effort into it! I'm so excited by Gravitek's progress that I've been building a generic drupalcamp website using Patterns for all configuration changes. It very well may be the world's first in-the-wild website pattern, and It's my hope that other camps can use this pattern to install a site for themselves in about ten minutes, then tweak it to their needs as desired. We'll see.
______
Senpai
Joel Farris | my 'certified to rock' score
Transparatech
http://transparatech.com
619.717.2805
YAML and Configuration Framework
On that note, a quick update is that YAML support has now been added to the Patterns module. It's a much nicer format to write and store patterns in so that they are easier to read and maintain. Check out the a simple YAML pattern sample here:
http://cvs.drupal.org/viewvc.py/drupal/contributions/modules/patterns/pa...
Also, we now have the initial code uploaded for the Configuration Framework here. It was a huge effort by sarvab and is basically a complete re-write of the Patterns core. It's has drastically simplified what's required to support different core and contrib modules by using XPATH. Check it out here:
http://drupal.org/project/configuration
http://drupal.org/cvs?commit=176462
See you at Drupalcon!
--
Gravitek Labs
--
Gravitek Labs
Are you planing a BoF or is
Are you planing a BoF or is there a session about the Configuration Framework, sounds like an interesting step forward and i would really like to see it taken for a spin by someone who knows what they are doing.
Also dont forget to put the browse CVS link on the project page ;)
a_c_m
BOFs and moving forward
We did have a couple of BOFs on the topic, not sure if you made it to one of them or not. We had some good discussions about the best way forward and the short of it is that we need to have some good hooks for import/export (or load/save, I/O, etc.) and have all modules support them. This would provide a common API for the setup, configuration or management of all modules and their settings.
In terms of Patterns and the Configuration Framework, we'd love to have a system by which we use APIs that are available and then fall back to using the Form API in cases where modules don't support what needs to be done.
Bhuga wrote up a thorough post about our discussions here: http://bhuga.net/2009/03/importexport-bof-drupalcon-2009-dc
Also, video is up from the presentation in case you missed it: http://www.archive.org/details/DrupalconDc2009-DrupalPatternsManagingAnd...
PS: thanks, the link for "browse cvs" has been added.
--
Gravitek Labs
--
Gravitek Labs
Well, my feeling is that
Well, my feeling is that Drupal must somehow seperate setup data from the other parts of the DB so that settings can be easily transfered. Today, everything is mixed.
Regards,
Sebastian
http://www.ariscommunity.com/
Yes, but it's it's complicated...
The question is, where you draw the line? It's a really grey area. For example is a view you create data or config? It should be considered configuration but some might argue that it's data in the database. Or, if you have a block with a little content in it, is that block config or data? Or even a node on a site for that matter, it could be considered part of the site config when used in certain ways. Obviously a node is a node and should be considered data but if you rely on it as part of some setup or display then it could be considered part of your configuration.
Or what about when you add a feed to your site with the Aggregator module. Again, is the feed (title and url) data or configuration?
Additionally modules can define static configuration items such as a menu item or view that can then be overridden. Is the item statically defined in code part of the configuration as well?
It would be great to be able to draw the line somewhere and define what's configuration vs. data in Drupal though I'm not sure how just yet. You could do it simply in that nodes, users and other lightweight items (feed items) are all data and everything else (including a feed for instance) should be considered configuration.
--
Gravitek Labs
--
Gravitek Labs
Maybe we are wrong?
You are right, it is indeed complicated. Therefore, if the approach is not working, we might have to look somewhere else. Basically, this problem was solved already many times in different use cases like synching your phone with your PC desktop calendar or address book. There, it works too, so it should be possible with a merge algorithm.
Regards,
Sebastian
http://www.ariscommunity.com/
It definitely can and will
It definitely can and will be done for sure, it just may take a while for a solution that resides in Drupal core. Have a look at some of the discussion going in the Packaging and Deployment group as well...
--
Gravitek Labs
--
Gravitek Labs
Pattern used for setting up similar sites?
Hin Senpai,
thanks for sharing your work report about the patterns module. If I understand that correctly it might be useful if I'm going to set up multiple sites with the same configuration. That is indeed important work, because similar stuff is used by system admins heavily to rollout changes to many hosts. It will be very interesting to see how that evolves.
http://www.ariscommunity.com/
Best practices to transport configurations changes
Instead of bookmarking this, I ask someone who visits this in a month to ping us and make us right u more thoroughly our approach at Agaric: using <a href="http://ceardach.com>Kathleen Murtagh's DBScripts module. (It is working and is documented, but can always use more detailed writing up.)
Our goal is for this to be complementary to Patterns and embedding configuration changes in modules (especially module update functions).
benjamin, Agaric Design Collective
benjamin, agaric
done
ping!
If you don't like how somethings being done...
I dug through all of the change management modules, patterns, dbscripts, etc in my quest for an easy, robust solution for managing ALL changes made through the web interface...
This includes all of the other modules who store data, like imagecache, custom_breadcrumbs, custom_links, custom_pagers, and on and on and on. Setting these things up is part of standard development of any website, not just config changes.
Inspired by journal.module and macro.module, I realized that almost all web-based changes are form submissions, and those can be captured and replayed with pretty good reliability in most cases. So I decided to set out to create a module to make it super easy to track updates and create an interface for exporting hook_update_N functions
After much deliberation with the other modules, I decided that form submission captures (macro.module) coupled with the site's custom module hook_update_N functions was a damn fine way to track what changes were made through the web interface.
The result? http://drupal.org/project/custom_module_tools
Why?
I've only just created this module this weekend, because I needed it. I will be using it as I finish up development on my current project and report back how things went.
Please check out the module, I'm really craving some feedback as well as suggestions as to what other tools we could include in this module package.
Jonathan Pugh
CareerNerd Industries
www.careernerd.com
Jon Pugh
Founder & CEO
THINKDROP
open source consulting
http://thinkdrop.net
hmm, I don't like my tone there :)
The title of my above comment is misleading, I was using the phrase tongue in cheek :)
I like the way things are being done, I just think that the edge cases are never addressed.
My module does not deal with Install profiles. It probably should/could integrate with the install_profile_api.
It could also be easily integrated with Deployment module, which I personally plan on doing soon.
Also, the module isn't necessarily the best way to migrate changes to certain things like views and panels, because those things are already in code.
However, it would be a fantastic way to track minor changes to views by multiple developers before the export is made.
Also, there are edge cases where things wouldn't work, like Node updates. unless the NIDs are in sync between dev and live, running the editform macro on a live site could update the wrong node.
Hopefully, with more R&D we can deal with all this.
Jonathan Pugh
CareerNerd Industries
www.careernerd.com
Jon Pugh
Founder & CEO
THINKDROP
open source consulting
http://thinkdrop.net
crazy idea
Wouldn't Database change control in sync with the form submissions (schema project) could be the line between code and data. By having form submissions tracked (devel macro?) associated with database changes (dbscripts?)
development->staging -> live (deployment module with drush .0/ ) : http://drupal.org/node/537488
It seems that deployment is the big problem and maybe the new install profile could set this all together and we could just go straight to live without fears.. I am trying to find a way and I can see each person trying to find a way somehow this should be though for Drupal and there should not be limitations since everything is open and nothing should be left behind.
my selection for startup is between, cvs_deploy, svn deploy, devel (macro), schema, dbscripts, custom_module_tools and drush and backup_migrate (for the files) this will be my first time ever will be sql dump every hour but we already know how it shouldn't be and I am afraid:
the n/v id's could be thought of an algorithm that would go back when necessary and kept order and sync,
Perhaps the odd/even issue could be handled with a checkbox (is there a patch for D6?)
I am already mixing the "must have" modules (features) in the develmachine and I can't seem to find out how to get this in different directories! How should I not mix the development modules with the staging server?
Kind Regards,
Jorge Nunes.