Configuration Management initiative and Drupal distributions

We encourage users to post events happening in the community to the community events group on https://www.drupal.org.
You are viewing a wiki page. You are welcome to join the group and then edit it. Be bold!

What considerations need to be addressed to ensure the Configuration Management Initiative is informed by and reflects the needs of Drupal distributions?

The Configuration Management Initiative tackles problems that are distinct from but overlap with what's done in Drupal 7 with the Features module, the tool of choice for producing distributions. Features allows capturing bundles of configuration and deploying them in a form that can be reproduced on multiple sites.

This wiki page aims to capture the specific distribution requirements of Drupal distributions and identify considerations and solutions required in the Configuration Management Initiative.

Bundles of related configuration

Problem statement

Building Drupal distributions often involves packaging up bundles or configuration relevant to a given use case. For example, configuration for an event use case might include an event content type and a calendar view.

Technical requirements

  • Dependency relationships between different configuration components are automatically detected. For example, an event content type is recognized as a dependency of an event calendar view.
  • Configuration components related to the use case are isolated from other configuration not related to the use case. For example, settings relevant to events (which might be owned by an arbitrary series of modules) are not mixed with other unrelated settings.

Drupal 7 contrib solutions

The Features module APIs enable bundling of configuration relevant to a use case and specifying of dependency relationships among components.

Drupal 8 CMI considerations

Currently neither of these technical requirements are met by CMI. There are no plans right now to support the first, and there is ongoing discussion about the second in terms of how granular the data stored in configuration files should be.

Outstanding CMI questions

Chained, persistent configuration overrides

Problem statement

It's often necessary to override distribution-based configuration in ways that persist through distribution updates. For example, a local site might require a different label for a menu item provided by a distribution.

Technical requirements

  • Customizations made to a distribution-provided configuration component can be stored locally. For example, a local site can change the title of a page display in a view provided by a distribution and store that change.
  • Changes to that component in future updates to the distribution will be merged with local changes. For example, a local site will receive a new display added to the view and also retain the local customization (page display title).
  • Persistent overrides can be chained. For example, a configuration component (e.g., view) provided by Distribution A is modified by Distribution B and then further customized on Site A. Future upgrades to Site A will receive customizations to the view from Distribution A and Distribution B while retaining its own customizations.

Drupal 7 contrib solutions

The Features Override 1.x module attempted to provide some relevant solutions, analyzing and capturing the difference between "default" (code-level base configurations as shipped with a distribution) and "normal" (local database-level overrides) components. A 2.x branch is being developed to try to address various shortcomings in 1.x.

Drupal 8 CMI considerations

CMI will provide an override capacity but it is mostly focused on managing site-specific override and there are no current plans to support the chained over-riding.

Outstanding CMI questions

Portable configuration not dependent on local data

Problem statement

Distributions must be able to provide configuration that is not dependent on specific local data structures.

Technical requirements

  • All configuration can be stored in ways that are independent of local data. For example, roles required for permissions captured in distributed configuration are identified without dependency on a local ID variable.

Drupal 7 contrib solutions

The Features APIs provide methods for identifying configuration components by machine name. The Features module provides special casing for Drupal core configuration types that lack portability.

Drupal 8 CMI considerations

CMI includes a focus on building UUID support in Drupal core. UUIDs are planned to provide a basis for portable configuration.

Outstanding CMI questions

Will configuration of core data structures such as user permissions be portable?

"Soft" vs. "hard" configuration

Problem statement

Distributions need to distinguish between "soft" configuration, which is likely to change on each site, and "hard" configuration, which is set in a persistent way.

Technical requirements

  • Certain configuration can be set in a non-persistent way while other configuration can be set in a peristant way.

Drupal 7 contrib solutions

Typically a distribution will handle configuration in two ways. Hard configuration goes into features as exportables. Soft configuration is done at install time through API calls like variable_set().

Drupal 8 CMI considerations

Outstanding CMI questions

Comments

Don't have time to fully

patcon's picture

Don't have time to fully digest this yet, but speaking of merging, perhaps we could take a cue from Chef's attribute merge strategy:
http://wiki.opscode.com/display/chef/Deep+Merge

Yes, I realize we're not using JSON, but thought it might be worth seeing how they operate :)

Thanks for writing this up nedjo! Hopefully I can contribute in a more useful fashion later

Interesting example

nedjo's picture

Many PHP frameworks seem to include support for inheritance in configuration files including merging and overriding. Some examples:

I have updated the wiki above

gdd's picture

I have updated the wiki above with the current status of most of these items in terms of CMI, and in general the answer is 'We are not supporting this and we have no plans to.' However I think this should be somewhat clarified.

When I took on this initiative, I defined a minimal set of functionality that would be necessary to have the project be a success. As you have noted, this is mostly focused on the staging/deployment workflow and doesn't inherently support what I have been referring to as a more 'vertical' workflow (supporting a specific distribution or feature.) This doesn't mean that these things aren't important or that I don't care about them, it just means that in a limited timeframe I think it is vital that we keep the feature set lean so we can get it implemented with some reasonable confidence. My mental model (based on TOTAL CONJECTURE and no facts at all) has been assuming a code freeze sometime in Q1 of 2012, so that has driven the scope of the project to some extent. If time permits, I hope to be able to iterate new features in on top of what gets committed in the first patches.

So all that said, what are the solutions for Features and distributions in Drupal 8 if only this minimal set of functionality makes it into Drupal 8?

To some extent things will be easier. All configuration will move through a single API and be stored in a standard file format in a single place, which will alone help immensely. However beyond that and some naming conventions, things aren't going to change a lot. Features will still have to know a lot about how the modules have implemented their configuration in order to work, and it is likely that contrib modules will still have to write some Features-specific code to glue things together while Features will have to include this glue code itself for core.

I have been thinking for a while that the granularity of data may cause some problems. For instance, in the patch, all the ssystem settings for performance are in one file. The easiest way for Features to pack those settings up would be to take a copy of the file, pull it into the Features module, then on install copy the file back over to the live config directory. However, as you note, this could cause issues in situations where you really only want to manage one setting without stomping all the others. An alternate option is that Features or an install profile grabs those individual settings and saves them and then writes them out through the API on install, but this will be some extra work. There has been some discussion about granular these files should be, but ultimately the options are a) one setting per file b) any one of a few organizational schemes that won't help this issue any. I am completely not sold on the one setting per file issue. This is, I believe, going to make a config directory that is completely unmanageable and has thousands of files, as well as making some things way more complicated than they need to be. Others disagree though so... discuss!

As far as the overrides, we plan on offering the ability to do local overrides and have those merge in with the master configuration (for things like database settings, api keys, etc). We don't have it in the current patch but it can be added later. We will also have some deep merging capability which will be necessary for the internationalization support. So while we have no plans to have any concept of overrides beyond that, it is something that should be manageable in contrib or followup patches. I had looked at many of the mentioned systems above, in particular Zend, before developing our system. However I didn't think inerited configuration was a compelling enough feature to make it critical for a version 1 feature set.

So this is where we're at, but I should also point out none of this is set in stone. If a lot of people think I'm on drugs in the priorities above, it can change. If someone wants to take charge of one of these features, that would be awesome! Keep the discussions going.

Thanks for the detailed response

nedjo's picture

I don't at all think that the CMI should try to meet all the requirements for distributions as sketched out above. But I do think it's critically important that the CMI is informed by and anticipates distribution requirements. We can't afford to revolutionize configuration management in a way that doesn't at least retain what's currently supported for distribution use cases. Ideally, in revolutionizing configuration management, we'll also greatly improve the core toolset that distribution functionality builds on. That mainly means that those of use involved in features and distribution development take the time to understand, participate in, and contribute to the CMI.

The starting place here is understanding whether and how the CMI patches are supporting the key relevant areas that already supported in Drupal core. These would include:

  1. Modules can ship with default configuration. In D7 core this includes a default image style. For variables, the defaults tend to be hard coded into variable_get() calls.
  2. Modules can alter or override the default configuration provided by other modules. In D7 core this includes a default image style that's altered through a hook_alter() implementation.
  3. Local sites can override this default configuration. In D7 core this includes creating an overridden image style.
  4. Local sites can revert to the default configuration. In D7 core this includes reverting an overridden image style.

Could you briefly point to or explain if and how each of these is handled in CMI?

I think the following

moshe weitzman's picture

I think the following represents the state of CMI in current patch:

  1. Modules can still ship with default config. They do so by shipping with a .config module. The current patch has one of these for image styles. This is what we used to think of as variable_get() default.

  2. Modules have to directly save to the Config store to override. There will be reactors to do so when config gets saved.

  3. Same as 2.

  4. You can already revert via code, and I'm sure there will be UI method as well. Might be provided by Contrib as the UX is not obvious.

Progress and issues

nedjo's picture

Thx Moshe or the clarifications.

Overall, having a uniform file-based configuration system should greatly facilitate the tasks Features tries to tackle in D7. No more need for totally disparate approaches for each type of configuration. No more need to try to coax so called "faux exportables" like variables and field instances to behave as if they supported file-based defaults.

But there also looks to be at least one significant potential regression here.

Modules have to directly save to the Config store to override.

This would mean that the current core functionality of programmatically altering existing default configuration would be lost. Presumably, reverting a piece of configuration would produce the original, unaltered default--not the default as programmatically altered.

I suppose as a workaround a module author could respond to an action triggered when a component is reverted, repeating the modification originally done at install time.

I'm not suggesting we need to keep the current drupal_alter() based approach. It's clumsy at best, mainly because there's no insight into what changes are being made; while the original default configuration can be well defined, the alter is a grab bag of whatever changes may have been made along the way.

Still, the loss of programmatic altering looks like a regression.

What might be better? I mused about this a few months ago but didn't follow up:

In D6 and D7 overrides are generally saved as forks. E.g., a default view is overridden in the database with a full copy of the view. Any further changes to the original are not passed to the override.

This approach creates a lot of challenges in updating distributed configuration. E.g., sites based on Features are customized via overrides and then cannot easily upgrade to later releases without losing the customizations.

As part of an improved configuration storage system, we should consider whether we can capture in overrides only the difference (additions, changes, and deletions)

Enabling local overrides to be structured from the start as differences to be deep merged rather than the full forks of D7 might help address many of the key challenges of distributing configuration. I have no idea, though, whether or how such an approach might fit with CMI.

I strongly believe that we

gdd's picture

I strongly believe that we can not let config be alterable through the tradition drupal_alter() mechanism. This would just be a complete disaster.

As for overrides, when we have talked about them, our hope has always been that we can make this mechanism work as diffs that get merged in. Our only currently planned level of overrides is that we would have a directory somewhere that would contain these diffs, for things like database information. When config is loaded, the main config is loaded and then these overrides get merged in. We have talked about more advanced situations where you have a hierarchy of overrides and merges that can happen. For instance, having a default config in sites/default with site-specific overrides in sites/ or the like. We could potentially do the same with profiles, the config they supply gets merged in. This really needs to be talked through carefully. The code has the potential to become extremely hairy and the hierarchy of which sets of config override which other sets needs to be really well defined.

Will you be in Denver? I'd be happy to be involved in a BOF or some other gathering to start talking through some of these issues.

Also I should point out there is no revert functionality built into the code at this point, although doing so isn't going to be horribly difficult (with some exceptions.)

Sounds great

nedjo's picture

our hope has always been that we can make [configuration overrides] work as diffs that get merged in ... We have talked about more advanced situations where you have a hierarchy of overrides and merges that can happen.

Obviously there's a lot of work left to be done, but overall if we could achieve these CMI would not only retain the basic override functionality of D7 but greatly improve the tools available to distros.

A note of caution re diffs. In Features Override 1.x diffs worked okay for many types of configuration, but not for all. Details at does not work for rules.

Added a section

nedjo's picture

on "soft" vs. "hard" configuration.

Report DC Kiev 2013

victor-shelepen's picture

Hello.

Drupal camp is appering. I prepare a report for. My topic is related with your initiatives.
http://camp13.drupal.ua/uk/sistema-upravleniya-konfiguraciey-v-drupal-8-...

It is in english - terms.
https://docs.google.com/document/d/1UZPosIiSGo_bGINcYbDnoZZl20nv7VRUIDsd...

My linkedin profile is.
http://www.linkedin.com/pub/viktor-likin/46/aa1/172

I search for people who can help me to clear this topic. I want to present my report to somebody who is in the initiative. I realy need your help. We will have a sprint too in Kyiv pefore the Drupal camp.

My skype login is vlikin.

Thank you.

Distributions

Group organizers

Group notifications

This group offers an RSS feed. Or subscribe to these personalized, sitewide feeds: