Configuration management architecture

You are viewing a wiki page. You are welcome to join the group and then edit it. Be bold!

This document represents the current state of the configuration management system for Drupal 8. Every attempt will be made to keep this document as up to date as possible.

[NOTE: Before the update that added this note, the page was last updated 2012-01-23. The page is by no means up to date.]

Overview

In a database-driven application such as Drupal, one of the most significant challenges for site builders is moving configuration and content changes between environments (dev, qa, production). Drupal has a number of mechanisms to support this currently—modules such as Views allow for exporting configuration in large chunks of PHP so it can be managed in version control systems, Image Styles allow defining new styles in code, the variable system allows storing individual pieces of data in a persistent way, and modules such as Features and Strongarm can automate exporting this functionality—but these methods are inconsistent across contributed modules and do not cover all instances.

The main goals of this part of the initiative are to improve configuration management by standardizing the format of configuration throughout core and creating an API allowing for better version control, history storage, and simplified management of all these things.

Overall Architecture

The proposed configuration system employs a three-level architecture:

Level 1: Signed file storage

At the lowest level, all configuration data will be stored on-disk using digitally signed XML files in a randomly named config directory (see "Security Considerations" below for more information.)

Individual configuration files (prefix.example.xml) will look like the following:

<?xml version="1.0"?>
<config>
  <some_string>Woo kittens!</some_string>
  <some_int>42</some_int>
  <some_bool>1</some_bool>
</config>

How these files are organized and named is still under discussion, however it is likely that the structure will vary quite a bit depending on use cases. Core might do files like core.site_information.xml, and something like Flag which other modules hook into to provide "discoverable" stuff might do both a module-specific configuration file module.flag.xml for "global" module settings, as well as provide the ability to do module.flag.$flagname.xml. At this point we will allow values to be nested, however this point is currently under some debate.

One advantage of storing configuration this way rather than the current system is that we can limit the data we load on each request; in the current system we have to load the entire contents of the variable table into memory on each page request.

Level 2: Active Store

This layer moves the configuration data from the file system to something that can be read and accessed much more readily. For the vast majority of Drupal sites this will be database storage, but for high-performance sites could be something like MongoDB or Redis. This is called the “active store.”

In the default configuration, this will push data into a central table called "config":

CREATE TABLE config (
   name varchar(255) NOT NULL DEFAULT '' COMMENT 'The identifier for the  configuration entry, such as module.example (the name of the file,  minus .xml).',
  data longtext NOT NULL COMMENT 'The raw XML data for this configuration entry.',
  PRIMARY KEY (name),
);

While at first glance the structure looks like the "variables" table in Drupal 7 and below, the fundamental difference is this table stores configuration objects (say, every site information setting: name, mission etc), while the variable stored single values (like the site name). Also, as said above, we are not loading the whole table in memory.

All site configuration data gets read out of this. The data here gets updated on two conditions:

  • UI changes (automatic): When the save button is clicked on an admin page, data gets written to both the active store and the .xml file from layer 1 (the digital signatures for the changed file gets regenerated).
  • Code changes (manual): When migrating configuration from dev to prod, for example, the underlying files will have changed, but the data in the database table will not. Data will continue to be read from the active store so that the site doesn't break. Site admins can replace the contents of the active store with the contents on disk via an administrative interface and/or a drush command.

Level 3: Configuration API

At this level are the actual API functions that module developers will interact with in order to manipulate configuration values; essentially, a replacement for variable_get()/variable_set().

// Load a set of configuration out of the active store.
// 'prefix.name' refers to the filename of the .xml file, without the extension.
$config = config('prefix.name');
// Access a single value out of the store.
echo $config->get('my.value');
// Change a value and save it back to both the active store and the filesystem.
$config->set('my.value','my new value');
$config->save();

Security Considerations

A bunch of talks were had with a great many members of the Drupal security team about the security of the web UI writing files directly to disk. Since these files may contain extremely sensitive data, such as database passwords, it's imperative that they not be readable by outside eyes, nor writable by rogue processes.

Here's what was figured out:

  • Within the root /sites/sitename/config/ there will be a .htaccess that does "Deny from all" (and IIS web.config that does the equivalent) so that these configuration files cannot be read at all, under normal conditions. Attempting to access the files directly will result in a 403 error.
  • In the root of the /sites/sitename/ there will be a read-only key.php file that will be written during installation, and contains a site-specific private key. This private key will be used to digitally verify that the contents of configuration files are not roguely changed on disk. Every .xml file will have a corresponding .sig file containing this signature.
  • If there's a mismatch between this hash and the file contents, the configuration system will refuse to load the file unless the administrator explicitly approves it. This will prevent someone from writing mischievous values to the file through some other exploit.
  • However, "just in case" this protection fails (or for web servers that do not support these override systems), all configuration files will be stored in a directory that is randomly named. This name is generated and the directory created at install time. The name will then be written to settings.php so that it is available for bootstrapping the config system. An API call will be available to get the current config directory.
  • There will also be a status check under admin/reports/status that throws flaming red warnings if .htaccess protection is turned off.
  • The goal here is "defense in depth", so an attacker would need to break through multiple layers in order to break anything undetected.

I'm a module developer. How do these proposed changes affect me?

variable_set()/variable_get() are going away

In Drupal 7 and below, all variables are global, so accessing and saving them is done this way:

// Load the site name out of configuration.
$site_name = variable_get('site_name', 'Drupal');
// Change the site name to something else.
variable_set('site_name', 'This is the dev site.');

In Drupal 8, configuration will only be lazy-loaded when needed. The above code would therefore change as follows:

// Load the site name out of configuration.
$config = config('core.site_information');
$site_name = $config->get('site_name');
// Change the site name to something else.
$config->set('site_name', 'My Awesome Site');
$config->save();

For "discoverable" chunks like entities or views, you can load all "sub-configuration" files (e.g. list of all views, list of all node types) with the following call (exact API call TBD): config_get_names_with_prefix('entity.node.');. This will return something like array('entity.node.page', 'entity.node.article');. So to retrieve all settings for the page node type run config('entity.node.page'). As a side note, this does mean that config('entity.node'); will not return anything (TBD whether it throws an exception).

Declaring your own configuration options

In the past, configuration was declared in a mish-mash of ways. system_settings_form() would automagically save all form elements as variables in the variable table. Modules like Flag and Views would employ a hook_X_default() type of pattern, and so on.

Under the new system, declaring your own "variable" style configuration options happens in XML files shipped with your module. These files will live in a ‘config’ directory that lives in your module directory. You declare the defaults for these variables in just this one place, as opposed to every time you try and retrieve them.

An example file might look like:

image.styles.thumbnail.xml:

<?xml version="1.0"?>
<config>
  <name>thumbnail</name>
  <effects>
    <image_scale>
      <data>
        <width>100</width>
        <height>100</height>
        <upscale>1</upscale>
      </data>
      <weight>0</weight>
    </image_scale>
  </effects>
</config>

During module installation, the configuration file will be copied to a user's config directory, and from then on the site-specific settings will be stored there. Similarly, when a module is disabled, these config settings would be removed from the live directory.

Defining default views, flags, etc. could be done much the same, but ultimately get stored in the active store and .xml files.

But my configuration data is way more complex than that!

Because we are using XML as a storage format, configuration objects can be arbitrarily complex. However we cannot currently support storing objects in the XML, only arrays. We will probably at some point in the future need to define an object storage schema for the XML. If your configuration data is especially complex, you're probably a fancy-pants developer and want to use a custom configuration class.

I'm a super-fancy-pants developer and need something more powerful than this. How do I override it?

The $config object will be a of a common class that provides the ->save() logic. For advanced use cases, it will be possible to override it, like so:

<?php
class MyAdvancedConfig extends DrupalConfig {
  function
someHelperMethod() {
   
$this->pants = 'fancy';
  echo
"Ma'am, you have my compliments on your fancy pants.";
  }
}
$config = config('core.site', 'MyAdvancedConfig');
$config->someHelperMethod();
$config->save();
?>

Migrating configuration from dev to prod

The overall workflow here would be as follows:

  1. On your development server, perform whatever configuration changes are needed through the UI. Create Views, check checkboxes, etc. These changes will get written to *both* the database table *and* the file system so the two are in sync (it will also re-generate the digital signatures of the underlying files).
  2. When finished with your edits, review the changes in the config directory with $vcs diff (or your tool of choice) to confirm they look as expected.
  3. If everything looks good, move the changed files to your production server in the usual way (SFTP, $vcs add/commit/push, $vcs update/pull). Nothing will immediately change, as your site will still be reading from the active store.
  4. Finally, go to admin/configuration/system/config or run drush config-update (note: final location/command likely different; making these up right now). This will outline the files that have changed. If the changes seem good, go ahead and confirm to overwrite the content of the active store with the on-disk configuration. The admin tool will offer to regenerate the file signatures if necessary.

Proposals under discussion

Performing action when new configuration is loaded

(NOTE: proposal only, not yet implemented, discussion at http://groups.drupal.org/node/190729)

There will be times when a module will need to perform some action when new configuration is loaded. For instance, when a new field is added, Drupal needs to create the appropriate tables and potential add instances of the field to bundles. This process will be handled as follows

  1. A reload of the active store from files is triggered. This can happen either through the admin UI or through an alternate mechanism like Drush.
  2. You are shown the list of files that have been changed, added, or deleted, and asked if you accept these changes.
  3. If you say yes, then the config system starts calling a hook in every module in the system, in order based on the directed acyclic graph (http://en.wikipedia.org/wiki/Directed_acyclic_graph) of the dependencies as defined in the module .info files. This should ensure that processes happen in the correct order. For instance, if a bundle has been created and then a field instance is added to that bundle, the bundle has to be created first.
  4. This function gets passed an object which points to the entirety of the new config. The old config can be accessed through the existing config system if need be.
  5. The modules look at the new config, compare to the old config in any way they care, and act based on it.
  6. When all this is complete (or potentially as it happens), the new config completely overwrites the old config in the active store.

Handling local overrides

(NOTE: proposal only, not yet implemented) It's often handy, for things like API keys or $db_url, to have site-specific configuration that is not checked into version control.

In Drupal 7 and below, you'd do the following in settings.php (or settings.local.php):

<?php $conf['site_name'] = 'This is the dev site'; ?>

Under the proposed system, there would overridden copies of any files that you need to override. When config is loaded into the active store, these values would get merged into the master version and written to the active store. For instance

/sites/default/config/core.site_information.xml

Within this file, would be particular keys/values that you're overriding on a per-value basis:

<?xml version=”1.0”?>
<config>
  <site_name>This is my cool overridden dev site name</site_name>
</config>

Internationalization

(NOTE: proposal only, not yet implemented, discussion at http://groups.drupal.org/node/185609)

Internationalization will be implemented by making every file language-specific, and making them contain only the configuration information that has actually been translated. So for instance you could have

site_information.en.xml (full set of configuration, shipped with Drupal)
site_information.se.xml (partially translated configuration, created install time)

Now say that your language is Swedish (.se). When files are loaded into the active store, the Swedish will be read and for any missing information the English version will be used. My understanding is that this is similar to how t() and the entity/field translation system works today. This information will then be the canonical set of configuration the site will run with. For configuration created by users on the fly (not shipped with modules or Drupal), the original version of the configuration will be the canonical version, and it can be in any language. Fallback will always happen to the original version of the configuration like entity/field translation.

Another thing that will need to be implemented is the ability to translate strings to an arbitrary language. This will be implemented as an additional parameter in the configuration API when you get() data from the config. By default it will use the site's current language (as extracted from the context system), however you will also have the option to pass in a specific language. In this case the system will read the configuration out of the appropriate file, or, if it doesn't exist there, out of the active store.

The way that Java/Android handle internationalized resources may be able to act as a model and/or inspiration

http://www.oracle.com/technetwork/java/javase/tech/techart-jsp-138681.html

There has also been discussion about the prospect of storing the language in the XML attributes. For instance

<site_information>
  <site_name attr="en">I am awesome!</site_name>
  <site_name attr="se">Jag är grymt!</site_name>
</site_information>

The advantage of this approach is that when you want to deploy for instance a new view, everything is stored in a single file. However, the disadvantage is that this data has much more complicated writing and merging requirements.

AttachmentSize
config_api.png48.83 KB

Comments

Very interesting and detailed

DjebbZ's picture

Very interesting and detailed explanations. Thanks heyrocker !

I have a few questions after reading this. Note that I haven't read any code.

1) Regarding the updating of the active store. It's supposed to happen through the UI or with a Drush command. It means that there's some common code executed between Drush and the form submit function. I'm wondering if there is/will be a real API to update it without going through the UI or using the Drush command. I ask this question because I remember Crell's DrupalCon sessions about Aphorisms of an API (there should be an API that is used through a UI, Drush and tests, so it's viable). I'm not giving lessons, just wandering if it's the case here. I'm not even sure using this API outside of Drush or the UI is useful, but as a practice the community often uses hook_update_N's to change configuration, this API may reveal necessary.

2) In the paragraph "Handling local overrides", I may miss something. Where do live the overridden copies of the config file ? In sites/{my site name}/config/{config file}.xml ? Is the settings.php still used to override specific values, that then get merged in sites/default/config/{config file}.xml ?

1) Of course there will be a

heyrocker's picture

1) Of course there will be a proper API in place so that any external process can interact with the system as needed.

2) This part is still under development, so I'm not sure where they will live. The hope is that settings.php will not serve this function anymore, and that its usage will become far more limited in Drupal 8.

on the overrides, how do you

meba's picture

on the overrides, how do you view developer complexity? D7 had 1 line to change variable, D8 needs a new file with a 5 line XML.

I wonder if there is an easier way? Maybe even a contrib 'Settings override' module that will expose such functionality to UI or drush command?

I dont think you are supposed

bfr's picture

I dont think you are supposed to write those XML's by hand(except for the default values of your own modules). Instead of variable_set() you now do $config->set() so that part has not changed that much if i understand correctly.

"there will be a .htaccess

willvincent's picture

"there will be a .htaccess that does "Deny from all" (and IIS web.config that does the equivalent)" as an nginx user, I'd respectfully like to request including details on the configuration of these rules for nginx in documentation somewhere. :)

Environment overrides

torrance123's picture

Will it be possible to specify a setting for one environment, eg. one for production and another for dev? There are all sorts of useful things for this, eg. having devel enabled and configured on dev, and disabled on live, or having logging set to show all on dev and disabled on live.

I wonder whether a new, optional attribute within the xml could specify this? eg.

<config>
  <module env="production">
    devel
  </module>
  <module>
    node
  </module>
</config>

...where if it is absent, the config is assumed to be for all environments, and if present, the config only applies to that environment. The environment would have be specified in settings.php, and would default to 'production' if absent.

I would presume this would be a dev-only option, editable only via manually editing the xml.

Production vs development vs

philippejadin's picture

Production vs development vs staging vs xyz is hard to manage : the wording and concepts are different for different use cases.

I think (I hope!) you could mimick this behavior using multiple sites :

  • For example on the production server, config lives in sites/example.com/config
  • A developper can work on a developement site with config stored in sites/example.localhost/config
  • Default (shared) settings, views etc, can be stored in sites/default/config

Once development is finished, developper moves/merge sites/example.localhost/config to sites/example.com/config

And commit/push the changes to the production server.

Moving config out of drupal structure

lambic's picture

It would be nice if there were a way to move the config dir to somewhere outside of drupal root, for those of us who don't completely trust .htaccess security.

Atomic transaction and two phase commit

dgoutam's picture

"UI changes (automatic): When the save button is clicked on an admin page, data gets written to both the active store and the .xml file from layer 1..."

I hope the following point would be considered as well in regards to the aove mentioned context :
- Atomic Transaction
- Two phase commit will be ensured with fall-back asynchronous notification process

In my opinion both of these are very difficult to implement but I do hope we could do this.

It will be very nice know if at all json was considered for this configuration option and if not the down-side of it.

Thanks for the detailed writeup.

s.Daniel's picture

I would also like to know if other formats besides XML have been considered. In the Java world people moved into "xml hell" and back out of it using annotations which of cause is not an option for us but json or YAML could be. They are easier to read for humans and use less storage than xml.

more explanation on signature strategy

jmcclelland's picture

I'm glad to see this work and thinking - a very important problem is being addressed and I look forward to it's implementation!

However, I'm unclear on the benefits of the strategy of signing configuration changes. It seems to add a significant amount of complexity in exchange for only marginally better security.

If I have write access to the config directory and I am able to write a configuration change to a file in that directory - it seems plausible that I will also have read-access to the necessary files needed to generate a valid signature.

Requiring the valid signature only seems to make it a more daunting programming task for the attacker.

I would prefer a simpler configuration implementation and a simpler direction for the site admin about how to secure your site (don't let rogue processes write to or read from your config directory). Remember - right now, if a rogue process can simply read your settings.php file your site most sites will be effectively compromised. Given that requirement for security, it seems reasonable to expect admins to be able to protect write access to the config directory as well.

I'm in favor of simple steps to add marginally better security (like lambic's suggestion to allow this config directory to be placed outside of the web root), but the signing approach seems like a lot of complexity for limited pay-off.

jamie

Please comment on this issue

heyrocker's picture

Please comment on this issue where we are already discussing removal of the file signing

http://drupal.org/node/1444620

Current Thinking

Barrett's picture

It appears this document has gotten behind the current thinking. Looking at what's in code for D8, it looks to me like:

  1. XML has been dropped in favor of YAML
  2. The naming convention for the files has moved away from prepending 'core.' or 'module.' and is using a format of <module>.<subgroup>.yml (e.g., 'system.rss.yml') for those modules that have groups of configuration or <module>.settings.yml for modules without groups of configuration (e.g., 'syslog.settings.yml')

I recognize this is still an evolving field, but is my understanding correct as of this moment?

I think you are

boztek's picture

I think you are correct.

There's also no db table for config at all - this stuff moves quick :)

Best new source of info?

pmackay's picture

Could this doc be updated, or could any links be posted that list a more up-to-date description of the new architecture and how it affects porting modules to D8?

Why the randomly named config folder?

csdco's picture

Obfuscating that seems a little bit silly. Why not just files/config? Things are protected via htaccess rules in just the same way that someone cannot view my sites/all/modules/some-file.php thanks to htaccess rules. It seems odd that only this specific directory is protected via naming obfuscation when other things aren't.

Also, can the location of the config directory be moved to outside of webroot? In settings.php it appears that you can only specify the folder name within files/.

short video on a cms according to ITIL

lbhopkins's picture

here is a short video describing how a Configuration Management System or CMS works according to ITIL, Configuration Management

Deployment & Build Systems & Change Management

Group organizers

Group categories

Tags

Group notifications

This group offers an RSS feed. Or subscribe to these personalized, sitewide feeds: