For the Drupal sites I am working on I have a setup using three different stages:
- R&D site - This one is for both testing contributed modules and my own development.
- Staging - A virtual copy of the live site to make sure that the things from the R&D site will work. Also used for testing updates/upgrades to core and contributed modules before they go live.
- Live site - Simply the live public site where all ends up after passing the two above.
This works pretty OK and straightforward. Lately though I have been thinking about way how this could be improved and made easier.
Everyone within this industry fully understands that separating code, design and content will in the long run be very beneficial when it comes to future scalability and maintenance.
But when it comes to us who are doing the behind the scenes work in adding new functionality it still adds quite a lot of extra work in regards to keeping a "log" when, for example, a new module is added, how it is configurated, changes needed to the rest of the system to make it work seamlessly in the existing environment.
First it has to be tested, if contrib module, or developed in R&D. Often this involves testing several similar modules and/or combinations before the wanted end result is found. I would say most developers does not have the discipline to keep a complete log here as most of it will simply not be needed as modules gets excluded due to that the functionality wasn't the expected.
Then, when the desired, functionality is found it needs to be tested on the Staging site with an up-2-date copy of the live site.
And lastly when it has passed all QA it will be implemented on the live site.
Especially going from the R&D to the Staging server will require quite a bit of reversed digging to make sure that all configuration for the new features are correctly applied. This is where it will be needed to compile the log to get the receipt for the installation on the live site later on.
Configuration/Staging APIs
This got me to think -What if could simply export and import the configuration of a Drupal installation. Not just the complete one, but also on module level.
Some modules, such as Views, has this and it is a big time saver.
I can see three basic different levels of configurations needed:
- Drupal Core - Just the configurations done to Drupal core.
- Modules - Configuration for individual modules, with dependencies.
- Content - Such as a View or a content type.
Then it should also be possible to use combinations of this. It would even be possible to generate a Site Profile from here that will include all settings needed to go from the first development to launching a new site.
When importing it the the new, live, site the UI will use a wizard that will guide through all the unique settings needed for the live site, such as connection to database, domain and all other things that is unique for an individual site in each module such as the code for the Google analytic module if that is used.
With a Configuration API I believe this would be possible.
The API could even be set up so you can connect two or more sites to be synced directly. This could be done in several ways:
- Staging -> Live - This would both synchronise code and configuration. If a new module/theme/etc exist on the staging server but not on the live it is automatically uploaded.
- Multiple Front Ends - Same as the above, but when the main Front End server is updated it will in turn update the others.
As the live site will have ongoing content changes it should also be possible to sync it back to the staging server. Then when new features are developed and needed to be tested before going live it would be a snap to update the staging server to mirror the current live site and all its content.
I do see that this would be a big big task to get implemented, mainly due to that every module will need to support it in one way or another to make it work.
With something like this I can see a lot of benefits in the usability of Drupal and it would also provide a grate base for creating installation profiles.
Comments
We are looking into the same problem
We are looking into the same problem. We have code updates that get made on the dev server, but our clients make content updates to staging as we build the sites out (there's simply too much content to add all at once when the development is complete).
This is completely mind boggling to me, though I see the reason we need to do it this way. There has to be some way to manage it, and I'm not seeing many solutions.
Some of the things I have been considering is to create triggers (either w/in D7 somehow or at a db level) to cascade content updates automatically from staging to dev as they occur. This might be possible if all content entities could somehow be flagged as content (even if it's a manual field applied to each content entity). Then, one could in theory take the updated content and fire off a process to update / add / delete the content on dev as we go - this has some real potential for errors if certain settings aren't synched up, but it could be possible I think.
What about workflow? How strong are the workflow managers for D7 with firing off external web services? In the past when I have worked with with enterprise CRM platforms, we could use workflow applications to route actions to external systems, or fire off an email for a content admin to review the change. I wonder if there is a way to take that sort of model, and when / if the admin approves the change, the workflow system also makes the change / adds the content on dev.
This is an over simplified idea, but what the hell, I'm spitballing here. We also have the capability for tracking content revisions within drupal. I wonder if there might be a way to at least track updates of existing content by enabling revision tracking on the staging machine and creating a module to compare the timestamps of the last revised (and approved / published) version on staging, and if the stamp is newer than the one on dev, trigger some sort of action to sync that entire node (and all revision data) back down to dev.
Since revision tracking is optional, one could simply enable this feature when the node type is created. This would be a pretty sloppy way of doing it, and would also come at a cost to systems where storage might be an issue.
One of the simpler options I have been considering is just flagging new content with a simple bitfield / boolean field that drupal can store for each content entity to indicate that a particular item has been flagged as updated. This could potentially allow a nightly process to find all the nodes with that bitfield set to 1, insert / update the records down to dev, and then reset it back to 0. It's not perfect but like I said, I'm just spitballing here. This is my first attempt at trying to solve this problem in a drupal context.
There's also a little voice in the back of my head whispering something about attaching a Globally unique id/hash to every entity that contains content, and having a regular process to index the GUID/UUID's attached to the records, and then run a process to compare them on the two servers. If the GUID/UUID is different, then update the record (but also make sure to copy that Unique hash back down to dev to prevent the process from being too resource intensive).
So many things to consider...
If you haven't already,
If you haven't already, consider using a Features-based development approach. This way, all configuration details are in code. Re-synchronizing your devel site with the content on the client's live site is as simple as downloading a database dump of the live site, then "updating" the site's features so that all the new configuration changes you've made in code are automatically put back in place into the database. When you go to update the client site, you don't touch their database directly; you just pull the new code onto their site and update the feature. It's almost an entirely different way of developing with Drupal, so it may take a while for you to be able to wrap your mind around it, but once you have the benefits of this approach are immediately evident.
Another side benefit of this is that if a client borks their site by playing with configuration settings they shouldn't have been playing with, it's trivial to reset the site's configuration back to a working state.
The Boise Drupal Guy!