Introduction
There was a meeting last week between heyrocker, webchick, and chrisstrahl to discuss the prospect of funding the CMI initiative. This meeting resulted in a funding proposal to resource this initiative through the Large Scale Drupal program and the Drupal community. This page and the attached documents are the results of the discussion. If you're interested in getting involved, please, contact me and I'll fill you in with further details of how to get involved.
Overview
In a database-driven application such as Drupal, one of the most significant challenges for site builders is moving configuration and content between environments (dev, QA, production), and these challenges can become extremely costly when added up over time over many site deployments. While there are contributed modules that provide workarounds for some of the current architectural limitations, they fall short in many areas. The goal of the Drupal 8 Configuration Management Initiative is to solve these problems in a holistic way, directly in Drupal core.
In Drupal 8, one of the strategic initiatives laid out by Dries is the Configuration Management initiative, lead by Greg "heyrocker" Dunlap, author of the Deploy module, who has a great deal of expertise in this space. The goal is to create a single, unified, system for managing configuration data, rather than disparate systems for variables vs. content types and views vs. other elements that have not yet been made "exportable," and ensure that when changes are made to configuration through Drupal’s user interface, they are represented in both the database and the file system.
A second prong to the initiative around content staging aims to provide consistency and unique, non-numeric identifiers to all entities in the system (users, content, etc.) in order to safely move content from one server to another, as well.
We are seeking funding through the Large Scale Drupal program to support this strategic initiative.
Current Limitations
In Drupal 7 and below, configuration changes are largely stored in and read from database tables. The advantage of this approach is it allows non-technical users to make changes to Drupal through web forms and see their changes take effect immediately. The disadvantage is that it’s extremely difficult to move configuration changes from one server to another, because the database also stores website content which you do not want to overwrite. Further, the distinction between whether something is configuration or content is not always clear; for example, taxonomy and menus can be both.
A variety of sub-optimal solutions have emerged to solve this problem, including:
- The “napkin” approach: Write down configuration changes done on dev, re-do them on live by hand. Tedious and error-prone.
- CTools exportables: Allows developers make their modules’ configuration exportable as source code, which can be moved from server to server. Only supported by certain modules.
- Features module: Provides a UI to export site configuration into a module that can be moved from server to server. Only supports certain modules.
- Deploy module: Allows moving content from one site to another by assigning each piece of content a unique, non-numeric identifier. Only supports certain types of content.
Future Directions
Because of limitations that exist within the core and contributed module space in Drupal 7, these improvements will only be applicable to Drupal 8. Changes of this nature affect underlying APIs, and it’s Drupal policy not to break backwards compatibility during stable releases. Because so much has already been built upon the existing Drupal 7 structure, any changes would necessitate major alterations to existing contributed projects.
This approach solves the following limitations:
- Configuration can be safely migrated between servers (and also placed under version control to allow for auditing and revert capabilities) because it’s stored on disk as files in addition to in the database.
- Content can safely be migrated between servers, because unique, non-numeric IDs can be compared to determine if the content already exists in the destination.
- Native support for configuration and content syndication in Drupal core; no add-on modules needed.
- Alternative, high-performance storage mechanisms such as Redis may also be used to store configuration data, and deployments can be managed with external deployment tools such as Capistrano.
In other words, this system will encompass the functionality of Features, CTools Exportables, and core configuration capabilities, in a unified system that works across all use cases.
Current Status
The initial patch to add the file-based configuration API has been committed to Drupal 8, and work is currently underway to convert existing configuration in Drupal core to use the system. A UUID library has also been committed to core, in order to support the moving of content between servers, although Drupal’s entity system does not yet use it.
Funding this initiative will provide both greater velocity and a core group of contributors that are able to advance development and support the work that Greg and Dixon are doing.
Funding Proposal
In order to support this important, strategic work, the Large Scale Drupal program is asking for the following resources:
- Funding for 15 hours per week of Greg Dunlap’s time over 4 months to do initiative leading, cat herding, and architectural reviews.
- Senior-level development resource for 5-10 hours a week for Y timeframe to help with architectural discussions and keep momentum moving.
- Solid developer resource for 10 hours a week for 4 months to assist with writing, reviewing, and testing patches.
- Funding to pay for required travel expenses to code sprints at the following major Drupal developer events: Drupal Dev Days Barcelona (June), pre-OSCON sprint at Acquia hours (July), and DrupalCon Munich (August) to facilitate coordination with multilingual, Views, entity and field API, and content staging teams.
Due to the nature of the community consensus-driven Drupal core development process, specific deliverables cannot be guaranteed. However, the following critical facets of configuration system work would be prioritized during this time.
Configuration Management
These items help address the biggest known current limitations with the system, including several release-blockers, denoted as “critical.” This work would provide the complete architectural vision behind the system so it could be properly vetted and tested (including an upgrade path from Drupal 7), and ensure that multilingual content and modules that provide complex data are supported.
- (critical) Configuration reload process (http://drupal.org/node/1447686): Provides the capability to load configuration data from disk back into the active store (e.g. pushing dev to live).
- (critical) Internationalization of configuration (http://drupal.org/node/1448330): Ensure that language-sensitive configuration can be stored and translated.
- (critical) Entity and field support (no issue link): Provide means of saving more complicated data structures to the configuration system.
- (critical) Views CMI support (no issue link): Provide support to the Views maintainers to convert Views module to CMI.
- (critical) Upgrade path for Drupal 7 (http://drupal.org/node/1464554, http://drupal.org/node/1500312): In order to ensure a smooth transition from Drupal 7 to Drupal 8, an upgrade path needs to be written for converted settings.
- (major) Determine configuration file format (http://drupal.org/node/1470824): Currently, the configuration system stores data in XML format, but limitations with this have been found. A workaround for these limitations or an alternate file format must be determined.
- (major) More performant data storage for active store (http://drupal.org/node/1500238): In order to ensure configuration data can be accessed as quickly as possible, alternative data storage mechanisms are being explored.
- (major) Re-architect object model in configuration system (no issue link) so performance enhancements can take place (e.g. static caching) for new backend storage methods.
- (normal) Support for system-wide conversions: Supporting the community in the conversion of various settings forms to the new configuration system throughout Drupal core.
Content Staging
The current plan for content staging is to provide the necessary underlying API support in Drupal core for this feature, but for the actual workflow pieces to be done with contributed modules, due to the diverse nature in requirements by different organizations.
- Entity API improvements (http://drupal.org/node/1346204): Convert entities into “real” classes, create a consitent, “lighter” storage mechanism for entity properties that are not fields, provide generic APIs applicable to all entities that are currently node-specific.
- Drupal 8 version of “Content Staging” module (http://drupal.org/node/1540656): Handles basic Dev -> Stg -> Live workflow for all core entities.
This funding would be used to get as far as possible on these items before DrupalCon Munich. Reporting to the LSD group would be done every week to keep abreast of progress.
Resources
Additional background reading, for the curious:
- Configuration Management Initiaitve group: http://groups.drupal.org/build-systems-change-management/cmi
- How to use the Drupal 8 configuration management system: http://heyrocker.com/how-use-drupal-8-configuration-system
- Original initiative announcement: http://buytaert.net/configuration-management-in-drupal-8, http://dev.nodeone.se/node/756
| Attachment | Size |
|---|---|
| image00.png | 63.7 KB |
| image01.png | 203.2 KB |
| LSD-CMI.pdf | 316.23 KB |
Comments
Good Start
It is a great idea to support multiple execution states contexts from a single drupal bootstrap or over various bootstraps, it will really make drupal dynamic, and reduce the need for tedeous exportation/importation of structural features to quite some extent.
"If you have to do a sequence of actions more than twice, they are in the wrong abstraction layer :D" - Learned from drupal
Will funding for one
Will funding for one initiative be a problem for other initiative owners? If I were an initiative owner (or other volunteer for that matter) I wonder would I want to spend my time for free while another initiative gets funding.
more business interest
There seem to be more "big business" interest in doing CMI and WSCCI work, or at least less grass-roots interest vs. the need of companies involved with LSD. Definitely if there are companies interested in sponsoring other initiatives, it would be great. LSD is interested in CMI and WSCCI.
In the Drupal 8 Multilingual Initiative, I get continued requests from people to get funding help to come to the Barcelona D8MI sprint and I have a relatively hard time finding sponsors for helping with food/soft-drink stuff for the sprint. Of course I don't have the power of Large Scale Drupal behind me, so my tools are postings like http://groups.drupal.org/node/221944.
The LSD funding does definitely seem more structured and ongoing vs. what we have seen before when Examiner or Acquia for example sponsored big chunks of Drupal 7 core work to help get the release out.
DA?
A good point was brought up on twitter: should the DA get involved in the funding?
DA and core development
From my limited understanding and some talks with Jacob Redding, the DA does not get involved in core development.
However, I may work with the DA to see if they would be an appropriate vehicle for tracking contributions. That's probably some sort of crazy legal quagmire, but it's worth talking about.
We did talk about this
Hey, good question.
We did think about this and discuss it a bit within the LSD program. Gabor did a good job talking about why the program chose to fund this aspect of core development, but I wanted to highlight a couple of other points.
First, Crell and heyrocker (the WSCCI and CMI initiative leads, respectively) certainly have and will continue to put forth a lot of their own time as a part of their duties - completely without pay. We can't fully fund these through the LSD group, and I'm not sure we would want to.
Secondly, we're providing funding to help accelerate the progress on existing initiative priorities. We're not setting the direction for the initiatives. This was a major factor in deciding to take these ones on as they're aligned with the goals of the LSD group. Our goal isn't to exclusively fund core development, and I certainly don't want to use funding as a way of influencing the priorities of the initiative leads.
Finally, anyone is welcome to put forth a proposal for funding of their projects. These gained attention, but that doesn't mean that others couldn't. Portions of core development (and lots of contrib) have been funded in the past by other companies.
Thanks everyone for taking
Thanks everyone for taking your time to answer my question, appreciate it. I am all for getting as much funding as we possibly can and I will be amongst the first to sponsor this initiative, since it's an important part of our daily work.
Funding for Drupal work is not new
This is a great question. Here's my opinion on it. For at least the last 6 years, maybe longer, Drupal development has been a mix of volunteer work and paid work. The exact mix has taken many different forms, and not worth getting into here, but I myself have put in a ton of volunteer time, and am also fortunate to currently be working at Acquia where I'm able to work on Drupal as my day job. Many volunteers put in a little bit of time, an hour here and an hour there, and there's many motivations and rewards for that: the joy of sharing something you know with others, being part of a community, building up your expertise, and more. A much smaller number of volunteers put in lots of time: say 10 hours a week or more, fairly consistently. Given the huge demand that currently exists for Drupal talent, all of these intensive contributors either already have (or if they've just recently started their period of intensive contribution to Drupal, will soon have) plenty of job opportunities to choose from. So it's already the case that people contributing lots of volunteer time to Drupal could be choosing to spend that time on paid work, but have their reasons for contributing volunteer time anyway (autonomy, working on something that's of interest to them, a personal desire to help solve a problem for which a funding source isn't available, and more).
Also, at least in this case, Greg was already being funded by his former employer, NodeOne, half-time for nearly a year to work on this initiative, and I've only heard positive things from people in the community about him having that opportunity, and gratitude towards NodeOne for creating it.
As an initiative owner, I
As an initiative owner, I just want to sound off here too. I certainly would never begrudge any other initiative owner getting some sort of sponsorship to work on their initiative. In a lot of ways I feel like the discuss is badly framed. Commerce Guys "SPONSORS" me 20% of my regular work week on my initiative. No one complains about that (and rightly so) because it would be really insane to do so. They don't have additional controls over what goes into core (all of my work is still vetted through the typical core contribution workflow, as is the case with ANY initiative) and any initiative owner having more time to work on their initiative just means we all benefit from a better Drupal 8. If a company see some benefit to a particular initiative, and want to provide some money to help that initiative succeed, that seems like a huge open source style win, not a point of contention.