There was a meeting last week between heyrocker, webchick, and chrisstrahl to discuss the prospect of funding the CMI initiative. This meeting resulted in a funding proposal to resource this initiative through the Large Scale Drupal program and the Drupal community. This page and the attached documents are the results of the discussion. If you're interested in getting involved, please, contact me and I'll fill you in with further details of how to get involved.
In a database-driven application such as Drupal, one of the most significant challenges for site builders is moving configuration and content between environments (dev, QA, production), and these challenges can become extremely costly when added up over time over many site deployments. While there are contributed modules that provide workarounds for some of the current architectural limitations, they fall short in many areas. The goal of the Drupal 8 Configuration Management Initiative is to solve these problems in a holistic way, directly in Drupal core.
In Drupal 8, one of the strategic initiatives laid out by Dries is the Configuration Management initiative, lead by Greg "heyrocker" Dunlap, author of the Deploy module, who has a great deal of expertise in this space. The goal is to create a single, unified, system for managing configuration data, rather than disparate systems for variables vs. content types and views vs. other elements that have not yet been made "exportable," and ensure that when changes are made to configuration through Drupal’s user interface, they are represented in both the database and the file system.
A second prong to the initiative around content staging aims to provide consistency and unique, non-numeric identifiers to all entities in the system (users, content, etc.) in order to safely move content from one server to another, as well.
We are seeking funding through the Large Scale Drupal program to support this strategic initiative.
In Drupal 7 and below, configuration changes are largely stored in and read from database tables. The advantage of this approach is it allows non-technical users to make changes to Drupal through web forms and see their changes take effect immediately. The disadvantage is that it’s extremely difficult to move configuration changes from one server to another, because the database also stores website content which you do not want to overwrite. Further, the distinction between whether something is configuration or content is not always clear; for example, taxonomy and menus can be both.
A variety of sub-optimal solutions have emerged to solve this problem, including:
- The “napkin” approach: Write down configuration changes done on dev, re-do them on live by hand. Tedious and error-prone.
- CTools exportables: Allows developers make their modules’ configuration exportable as source code, which can be moved from server to server. Only supported by certain modules.
- Features module: Provides a UI to export site configuration into a module that can be moved from server to server. Only supports certain modules.
- Deploy module: Allows moving content from one site to another by assigning each piece of content a unique, non-numeric identifier. Only supports certain types of content.
Because of limitations that exist within the core and contributed module space in Drupal 7, these improvements will only be applicable to Drupal 8. Changes of this nature affect underlying APIs, and it’s Drupal policy not to break backwards compatibility during stable releases. Because so much has already been built upon the existing Drupal 7 structure, any changes would necessitate major alterations to existing contributed projects.
This approach solves the following limitations:
- Configuration can be safely migrated between servers (and also placed under version control to allow for auditing and revert capabilities) because it’s stored on disk as files in addition to in the database.
- Content can safely be migrated between servers, because unique, non-numeric IDs can be compared to determine if the content already exists in the destination.
- Native support for configuration and content syndication in Drupal core; no add-on modules needed.
- Alternative, high-performance storage mechanisms such as Redis may also be used to store configuration data, and deployments can be managed with external deployment tools such as Capistrano.
In other words, this system will encompass the functionality of Features, CTools Exportables, and core configuration capabilities, in a unified system that works across all use cases.
The initial patch to add the file-based configuration API has been committed to Drupal 8, and work is currently underway to convert existing configuration in Drupal core to use the system. A UUID library has also been committed to core, in order to support the moving of content between servers, although Drupal’s entity system does not yet use it.
Funding this initiative will provide both greater velocity and a core group of contributors that are able to advance development and support the work that Greg and Dixon are doing.
In order to support this important, strategic work, the Large Scale Drupal program is asking for the following resources:
- Funding for 15 hours per week of Greg Dunlap’s time over 4 months to do initiative leading, cat herding, and architectural reviews.
- Senior-level development resource for 5-10 hours a week for Y timeframe to help with architectural discussions and keep momentum moving.
- Solid developer resource for 10 hours a week for 4 months to assist with writing, reviewing, and testing patches.
- Funding to pay for required travel expenses to code sprints at the following major Drupal developer events: Drupal Dev Days Barcelona (June), pre-OSCON sprint at Acquia hours (July), and DrupalCon Munich (August) to facilitate coordination with multilingual, Views, entity and field API, and content staging teams.
Due to the nature of the community consensus-driven Drupal core development process, specific deliverables cannot be guaranteed. However, the following critical facets of configuration system work would be prioritized during this time.
These items help address the biggest known current limitations with the system, including several release-blockers, denoted as “critical.” This work would provide the complete architectural vision behind the system so it could be properly vetted and tested (including an upgrade path from Drupal 7), and ensure that multilingual content and modules that provide complex data are supported.
- (critical) Configuration reload process (http://drupal.org/node/1447686): Provides the capability to load configuration data from disk back into the active store (e.g. pushing dev to live).
- (critical) Internationalization of configuration (http://drupal.org/node/1448330): Ensure that language-sensitive configuration can be stored and translated.
- (critical) Entity and field support (no issue link): Provide means of saving more complicated data structures to the configuration system.
- (critical) Views CMI support (no issue link): Provide support to the Views maintainers to convert Views module to CMI.
- (critical) Upgrade path for Drupal 7 (http://drupal.org/node/1464554, http://drupal.org/node/1500312): In order to ensure a smooth transition from Drupal 7 to Drupal 8, an upgrade path needs to be written for converted settings.
- (major) Determine configuration file format (http://drupal.org/node/1470824): Currently, the configuration system stores data in XML format, but limitations with this have been found. A workaround for these limitations or an alternate file format must be determined.
- (major) More performant data storage for active store (http://drupal.org/node/1500238): In order to ensure configuration data can be accessed as quickly as possible, alternative data storage mechanisms are being explored.
- (major) Re-architect object model in configuration system (no issue link) so performance enhancements can take place (e.g. static caching) for new backend storage methods.
- (normal) Support for system-wide conversions: Supporting the community in the conversion of various settings forms to the new configuration system throughout Drupal core.
The current plan for content staging is to provide the necessary underlying API support in Drupal core for this feature, but for the actual workflow pieces to be done with contributed modules, due to the diverse nature in requirements by different organizations.
- Entity API improvements (http://drupal.org/node/1346204): Convert entities into “real” classes, create a consitent, “lighter” storage mechanism for entity properties that are not fields, provide generic APIs applicable to all entities that are currently node-specific.
- Drupal 8 version of “Content Staging” module (http://drupal.org/node/1540656): Handles basic Dev -> Stg -> Live workflow for all core entities.
This funding would be used to get as far as possible on these items before DrupalCon Munich. Reporting to the LSD group would be done every week to keep abreast of progress.
Additional background reading, for the curious:
- Configuration Management Initiaitve group: http://groups.drupal.org/build-systems-change-management/cmi
- How to use the Drupal 8 configuration management system: http://heyrocker.com/how-use-drupal-8-configuration-system
- Original initiative announcement: http://buytaert.net/configuration-management-in-drupal-8, http://dev.nodeone.se/node/756