Configuration management architecture

This document DOES NOT represent the current state of the configuration management system for Drupal 8. No attempt has been made to keep this document as up to date as possible.

STRONGLY ADVISE NOT USING THIS DOCUMENT -- Unless you are looking for history on D8 early development.

By the same token, any link you followed to reach this document is also out of date and should be corrected to point elsewhere.

[NOTE: Before the update that added this note, the page was last updated 2012-01-23. The page is by no means up to date.]

Overview

In a database-driven application such as Drupal, one of the most significant challenges for site builders is moving configuration and content changes between environments (dev, qa, production). Drupal has a number of mechanisms to support this currently—modules such as Views allow for exporting configuration in large chunks of PHP so it can be managed in version control systems, Image Styles allow defining new styles in code, the variable system allows storing individual pieces of data in a persistent way, and modules such as Features and Strongarm can automate exporting this functionality—but these methods are inconsistent across contributed modules and do not cover all instances.

The main goals of this part of the initiative are to improve configuration management by standardizing the format of configuration throughout core and creating an API allowing for better version control, history storage, and simplified management of all these things.

Overall Architecture

The proposed configuration system employs a three-level architecture:

Level 1: Signed file storage

At the lowest level, all configuration data will be stored on-disk using digitally signed XML files in a randomly named config directory (see "Security Considerations" below for more information.)

Individual configuration files (prefix.example.xml) will look like the following:

How these files are organized and named is still under discussion, however it is likely that the structure will vary quite a bit depending on use cases. Core might do files like core.site_information.xml, and something like Flag which other modules hook into to provide "discoverable" stuff might do both a module-specific configuration file module.flag.xml for "global" module settings, as well as provide the ability to do module.flag.$flagname.xml. At this point we will allow values to be nested, however this point is currently under some debate.

One advantage of storing configuration this way rather than the current system is that we can limit the data we load on each request; in the current system we have to load the entire contents of the variable table into memory on each page request.

Level 2: Active Store

This layer moves the configuration data from the file system to something that can be read and accessed much more readily. For the vast majority of Drupal sites this will be database storage, but for high-performance sites could be something like MongoDB or Redis. This is called the “active store.”

In the default configuration, this will push data into a central table called "config":

While at first glance the structure looks like the "variables" table in Drupal 7 and below, the fundamental difference is this table stores configuration objects (say, every site information setting: name, mission etc), while the variable stored single values (like the site name). Also, as said above, we are not loading the whole table in memory.

All site configuration data gets read out of this. The data here gets updated on two conditions:

• UI changes (automatic): When the save button is clicked on an admin page, data gets written to both the active store and the .xml file from layer 1 (the digital signatures for the changed file gets regenerated).
• Code changes (manual): When migrating configuration from dev to prod, for example, the underlying files will have changed, but the data in the database table will not. Data will continue to be read from the active store so that the site doesn't break. Site admins can replace the contents of the active store with the contents on disk via an administrative interface and/or a drush command.

Level 3: Configuration API

At this level are the actual API functions that module developers will interact with in order to manipulate configuration values; essentially, a replacement for variable_get()/variable_set().

Security Considerations

A bunch of talks were had with a great many members of the Drupal security team about the security of the web UI writing files directly to disk. Since these files may contain extremely sensitive data, such as database passwords, it's imperative that they not be readable by outside eyes, nor writable by rogue processes.

Here's what was figured out:

• Within the root /sites/sitename/config/ there will be a .htaccess that does "Deny from all" (and IIS web.config that does the equivalent) so that these configuration files cannot be read at all, under normal conditions. Attempting to access the files directly will result in a 403 error.
• In the root of the /sites/sitename/ there will be a read-only key.php file that will be written during installation, and contains a site-specific private key. This private key will be used to digitally verify that the contents of configuration files are not roguely changed on disk. Every .xml file will have a corresponding .sig file containing this signature.
• If there's a mismatch between this hash and the file contents, the configuration system will refuse to load the file unless the administrator explicitly approves it. This will prevent someone from writing mischievous values to the file through some other exploit.
• However, "just in case" this protection fails (or for web servers that do not support these override systems), all configuration files will be stored in a directory that is randomly named. This name is generated and the directory created at install time. The name will then be written to settings.php so that it is available for bootstrapping the config system. An API call will be available to get the current config directory.
• There will also be a status check under admin/reports/status that throws flaming red warnings if .htaccess protection is turned off.
• The goal here is "defense in depth", so an attacker would need to break through multiple layers in order to break anything undetected.

I'm a module developer. How do these proposed changes affect me?

variable_set()/variable_get() are going away

In Drupal 7 and below, all variables are global, so accessing and saving them is done this way:

In Drupal 8, configuration will only be lazy-loaded when needed. The above code would therefore change as follows:

For "discoverable" chunks like entities or views, you can load all "sub-configuration" files (e.g. list of all views, list of all node types) with the following call (exact API call TBD): config_get_names_with_prefix('entity.node.');. This will return something like array('entity.node.page', 'entity.node.article');. So to retrieve all settings for the page node type run config('entity.node.page'). As a side note, this does mean that config('entity.node'); will not return anything (TBD whether it throws an exception).

Declaring your own configuration options

In the past, configuration was declared in a mish-mash of ways. system_settings_form() would automagically save all form elements as variables in the variable table. Modules like Flag and Views would employ a hook_X_default() type of pattern, and so on.

Under the new system, declaring your own "variable" style configuration options happens in XML files shipped with your module. These files will live in a ‘config’ directory that lives in your module directory. You declare the defaults for these variables in just this one place, as opposed to every time you try and retrieve them.

An example file might look like:

During module installation, the configuration file will be copied to a user's config directory, and from then on the site-specific settings will be stored there. Similarly, when a module is disabled, these config settings would be removed from the live directory.

Defining default views, flags, etc. could be done much the same, but ultimately get stored in the active store and .xml files.

But my configuration data is way more complex than that!

Because we are using XML as a storage format, configuration objects can be arbitrarily complex. However we cannot currently support storing objects in the XML, only arrays. We will probably at some point in the future need to define an object storage schema for the XML. If your configuration data is especially complex, you're probably a fancy-pants developer and want to use a custom configuration class.

I'm a super-fancy-pants developer and need something more powerful than this. How do I override it?

The $config object will be a of a common class that provides the ->save() logic. For advanced use cases, it will be possible to override it, like so:

Migrating configuration from dev to prod

The overall workflow here would be as follows:

1. On your development server, perform whatever configuration changes are needed through the UI. Create Views, check checkboxes, etc. These changes will get written to *both* the database table *and* the file system so the two are in sync (it will also re-generate the digital signatures of the underlying files).
2. When finished with your edits, review the changes in the config directory with $vcs diff (or your tool of choice) to confirm they look as expected.
3. If everything looks good, move the changed files to your production server in the usual way (SFTP, $vcs add/commit/push, $vcs update/pull). Nothing will immediately change, as your site will still be reading from the active store.
4. Finally, go to admin/configuration/system/config or run drush config-update (note: final location/command likely different; making these up right now). This will outline the files that have changed. If the changes seem good, go ahead and confirm to overwrite the content of the active store with the on-disk configuration. The admin tool will offer to regenerate the file signatures if necessary.

Proposals under discussion

Performing action when new configuration is loaded

(NOTE: proposal only, not yet implemented, discussion at http://groups.drupal.org/node/190729)

There will be times when a module will need to perform some action when new configuration is loaded. For instance, when a new field is added, Drupal needs to create the appropriate tables and potential add instances of the field to bundles. This process will be handled as follows

1. A reload of the active store from files is triggered. This can happen either through the admin UI or through an alternate mechanism like Drush.
2. You are shown the list of files that have been changed, added, or deleted, and asked if you accept these changes.
3. If you say yes, then the config system starts calling a hook in every module in the system, in order based on the directed acyclic graph (http://en.wikipedia.org/wiki/Directed_acyclic_graph) of the dependencies as defined in the module .info files. This should ensure that processes happen in the correct order. For instance, if a bundle has been created and then a field instance is added to that bundle, the bundle has to be created first.
4. This function gets passed an object which points to the entirety of the new config. The old config can be accessed through the existing config system if need be.
5. The modules look at the new config, compare to the old config in any way they care, and act based on it.
6. When all this is complete (or potentially as it happens), the new config completely overwrites the old config in the active store.

Handling local overrides

(NOTE: proposal only, not yet implemented) It's often handy, for things like API keys or $db_url, to have site-specific configuration that is not checked into version control.

In Drupal 7 and below, you'd do the following in settings.php (or settings.local.php):

<?php $conf['site_name'] = 'This is the dev site'; ?>

Under the proposed system, there would overridden copies of any files that you need to override. When config is loaded into the active store, these values would get merged into the master version and written to the active store. For instance

/sites/default/config/core.site_information.xml

Within this file, would be particular keys/values that you're overriding on a per-value basis:

Internationalization

(NOTE: proposal only, not yet implemented, discussion at http://groups.drupal.org/node/185609)

Internationalization will be implemented by making every file language-specific, and making them contain only the configuration information that has actually been translated. So for instance you could have

Now say that your language is Swedish (.se). When files are loaded into the active store, the Swedish will be read and for any missing information the English version will be used. My understanding is that this is similar to how t() and the entity/field translation system works today. This information will then be the canonical set of configuration the site will run with. For configuration created by users on the fly (not shipped with modules or Drupal), the original version of the configuration will be the canonical version, and it can be in any language. Fallback will always happen to the original version of the configuration like entity/field translation.

Another thing that will need to be implemented is the ability to translate strings to an arbitrary language. This will be implemented as an additional parameter in the configuration API when you get() data from the config. By default it will use the site's current language (as extracted from the context system), however you will also have the option to pass in a specific language. In this case the system will read the configuration out of the appropriate file, or, if it doesn't exist there, out of the active store.

The way that Java/Android handle internationalized resources may be able to act as a model and/or inspiration

http://www.oracle.com/technetwork/java/javase/tech/techart-jsp-138681.html

There has also been discussion about the prospect of storing the language in the XML attributes. For instance

The advantage of this approach is that when you want to deploy for instance a new view, everything is stored in a single file. However, the disadvantage is that this data has much more complicated writing and merging requirements.