Configuration management sprint followup discussion

We encourage users to post events happening in the community to the community events group on https://www.drupal.org.
gdd's picture

First I want to thank everyone for their feedback in the last thread about the configuration management system. While it was long and hard, a lot of really good ideas came out of it. What I'd like to do here is summarize some of the points of contention and propose alternatives for further discussion. It seems there is a lot of agreement around the basic ideas behind the proposal and if we can work out the controversial pieces, then we are in pretty good shape to move forward. If you need the background to this discussion, you can read the original proposal as well as the very long comment thread it spawned (second longest in gdo history!)

So first off, I think it is fair to say that the community hates the proposed json.php hybrid format with the burning passion of a thousand suns. This was meant to be a compromise which addressed the concerns of having sensitive information in a readable directory that could be exposed if .htaccess/web.config protection was not present or configured improperly. The major objections to this are

  1. It creates another 'Drupalism', a format that is similar to an established standard with our own special sauce added (just like .info files are sort-of-but-not-really .ini files.)
  2. It significantly detracts from developer experience by breaking syntax highlighting and code verification in IDEs and editors.

Out of that thread came two related ideas that, when put together, should hopefully allow us to use pure JSON as well as address any security concerns that arise.

The first idea was put forth by grendzy, who suggested that instead of storing the config files in a standard place, we store them in a directory that is named using a hash of your site's private key, like sites/default/config_723fd490de3fb7203c3a408abee8c0bf3c2d302392. The files in this directory would still be protected via .htaccess/web.config, but if that protection failed then the files would still be essentially impossible to find. This means we could store pure, native .json files eveywhere instead, to still bring the benefits of JSON (human editable, syntax checkable, interoperability with external configuration management tools, native + speedy encoding/decoding functions), without the confusing and controversial PHP wrapper.

The thread discussing this idea has some more commentary on this solution. It seems like this idea has a lot of support, especially from other security team members. But I've asked for a more formal ruling before we get too married to it; I don't want to end up in a situation where we start building a system and then discover it isn't going to fly.

The second idea was to make the actual writing of the files optional. In the original proposal, anytime you saved your configuration in the UI, the data would be saved both to the Active Store (the database by default) and to the .json.php files. These files are really only used for deployment, and the only time they get accessed is when new data is being saved or when they have been pushed to a new server (or in some rare cases when a developer edits them by hand). If we make the default that they never get written, then the security of having .json files is greatly mitigated. We would provide a mechanism so that non-technical users could have an easy export/import process (for instance chx has suggested providing a direct zip upload/download mechanism). We would also still offer people the opportunity to turn it on by default if they want. Doing this would trigger a hook_requirements() check to verify that .htaccess protection is working, and if not we could either block it or put up a big flashing red warning of death about how taking this action could kill babies or whatever. On top of all this, the location of the files would still be configurable, which would enable developers to locate them outside of the webroot (which is actually the reccomended scenario, just like with private files).

So I would love to see some more discussion of these topics. What won't work here? What are we missing? Thanks everyone.

Comments

Thank you, good

pounard's picture

Thank you, good synthesis.

The folder named with a hash will make site deployment scripting really hard to achieve. Drush can help, but Drush is not the only scripting you would want to do in the end.

or in some rare cases when a developer edits them by hand

I disagree with this statement, I think more people than you think actually set up a lot of configuration bits using directly settings.php file. I really think the JSON format is non writable-friendly (I don't see why actual INI files are no good here, easier to read and write, supports hierarchy and more). Plus the JSON format doesn't accept comments, and that's something that really worth to exists.

I think the DX will be more in the final API that in the files, but the best AX (Admin experience) will be fulfilled only if you allow them to fully configure environmental and misc. performance settings without having to click in the UI at all, I mean, by editing fully self-documented configuration files (or providing self-documented configuration files sample).

That's only an opinion, but I really think this point is really important.

PS: This doesn't solve the environmental override, or site modes (devel, prod, etc..) problem, and I think this problem is linked to either the chosen file format either in the file system hierarchy and should be treated within this same discussion.

Pierre.

The proposed solution is

Sylvain Lecoy's picture

The proposed solution is really not maintainable, let me explain why:

  1. you have to find out what hash correspond to what. OEI536EHD63F65DX8D.json.php is not explicit.
  2. Having a <?php die(); in the JSON representation is a "don't modify this line its set in stone"
  3. Having a hash is a supplementary step to config maintainers. It adds complexity for config deployment or change

The proposition of using PHP seems obfuscated for reasons that I don't understand (maybe commercial ?).

No

gdd's picture

1) We are not talking about hashing the file names, just the name of the directory they are contained in.

2) The whole point of this new proposal is that we won't be using json.php anymore, we would just be using pure JSON, so this also doesn't apply at all.

3) There will surely be an API call which can get the current configuration directory and would make it quite simple for deployment scripts to be written. In fact, in the proposal, as long as you can access the site's key, you can calculate this yourself in you deployment scripts.

Please re-read the current proposal so that you can fully understand it.

Thanks to enlighten me

Sylvain Lecoy's picture

Thanks to enlighten me heyrocker, what about the API ? This is maybe the most important part that you are maybe missing in your résumé.

Another is why JSON instead of PHP but maybe this one is a taboo..?

There is a lot of discussion

gdd's picture

There is a lot of discussion of PHP vs JSON in the first thread, and it seems to me that the vast majority of posters were in favor of JSON as long as it wasn't a JSON/PHP hybrid. The API is still under construction, but what there is of it is described in the first post under 'Configuration API' and the section on how this affects module developers. I'm happy continuing discussion on these issues, but I would prefer to have it happen in a new thread so that we can keep this thread focused on a solution to the one problem that was easily the most controversial, which is currently my biggest concern.

Comments

sun's picture

While comments in configuration would obviously get lost when regenerating + writing them from scratch, I indeed can imagine that comments in configuration can be helpful and desired.

Possibly odd example, but it's one concrete I know of:

<?php
$view
= new view;
$view->disabled = FALSE; /* Edit this to true to make a default view disabled initially */
?>

Daniel F. Kudwien
netzstrategen

Scripting

chx's picture

cd sites/default/conf/$(ls sites/default/conf|head -1)

Feel free to port to PHP :)

This is my personal opinion

meba's picture

This is my personal opinion as a member of the Security team, not an opinion of the whole team...

I would potentially be OK with the proposal of not writing by default and writing to a unique directory hash but I am still afraid it will still create too much space for error. Greg, what about just requiring that the config directory is outside webroot? I have a feeling that people not having access to any directory outside webroot will be on cheap shared FTP only hosting providers only and those are not the people using configuration management files anyway.

There have been several

gdd's picture

There have been several people who are vocal opponents of this, most notably Larry. Some of my colleagues here at NodeOne have mentioned we have had clients on web hosts that did not allow anything outside webroot (note that just because a site is on cheap shared hosting doesn't mean an experienced developer isn't still stuck figuring out a way to maintain it.) So while personally I still don't think that requiring this is a realistic option, we should definitely encourage it as much as possible. I would love to hear more discussion around this from people with different experiences though.

Some of my colleagues here at

q0rban's picture

Some of my colleagues here at NodeOne have mentioned we have had clients on web hosts that did not allow anything outside webroot

So if that's the case, you write your own storage mechanism, or use a mechanism provided by Contrib. I don't think ongoing it should be Core's responsibility to try to be all things to all hosts. ;) It should provide simple, secure solutions, and allow you to modify those solutions.

I've worked for .edu clients

Josh Benner's picture

I've worked for .edu clients that have very locked-down hosting, including in at least one instance, a host that did not allow storage outside webroot.

Test if the

perusio's picture

server can write outside the webroot. If not then signal a problem within hook_requirements() and fallback to the bad idea of hashed dirnames inside the webroot. As much as I dislike this idea is better than the .json.php and die() on the config file.

I've said it before, that trying to make the least common denominator (shared hosting/locked down environments) be the rule is a bad idea. It degrades the DX and also in terms of sysadmin/devops.

Make it optional. Give the developer/site mantainer the option of deciding which profile he prefers. Summarizing:

  1. Drupal bootstraps, tries to write the config files outside the webroot.

  2. It can't write. This can happen, because:

    • The environment is shared hosting.

    • The site is run as a unprivileged user with all dirs inside the webroot being owned by this user and the www-data user (to use a debianism).

    • The environment is locked down due to "security policies" in place.

  3. Stop the boostrap and present a user with a choice:

    • Fix the permissions if applicable/possible and use the advanced profile.

    • Proceed with the default profile: config files are written inside webroot as JSON files with hashed dirnames.

  4. Resume the boostrap and finish.

Like Benjamin says below. I think that making the files be directly editable is something to be wished devoutly. After all we're doing this because having the config in the DB introduces a lot of issues when developing/managing a site and because the currently available solutions, namely features, are lacking in a lot of places.

I don't like either any of

pounard's picture

I don't like either any of the solutions proposed, but I do agree with you, hash in the directory in the less worst.

I hope that by bootstrap you mean install phase, because you cannot allow normal bootstrap to attempt file read/write every hit, it'll be like a performance killer if the file system is slow.

Pierre.

Yes

perusio's picture

you're right. This is something that is done once not something that is to be done for every boostrap. In fact

<?php
 
/**
    * Implements hook_requirements()
   */
  
(...)
     if (
$phase == 'install') {
       (...)
// do the config checking stuff here.
    
}
?>

Thanks for pointing out my loose usage of the drupal loaded term bootstrap.

I am a fan of the latter idea

q0rban's picture

I am a fan of the latter idea, keeping it optional.

Doing this would trigger a hook_requirements() check to verify that .htaccess protection is working, and if not we could either block it or put up a big flashing red warning of death about how taking this action could kill babies or whatever. On top of all this, the location of the files would still be configurable, which would enable developers to locate them outside of the webroot (which is actually the reccomended scenario, just like with private files).

I think one of the great benefits of having it optional that is being missed here is having core require a location outside the docroot. Then we don't have to even mess with htaccess or any other checks. We make the storage mechanism pluggable so that they could use whatever storage method they choose, provided they either write their own storage mechanism, or use a contrib solution for another mechanism (File store inside the docroot, DropBox, MongoDB, Twitter (jk)).

So, on a higher level, we have core define the extensible Configuration API (Level 3). The actual storage mechanism is pluggable, and by default uses db store (Level 2 only). Core also ships with a storage mechanism that introduces a second layer of storage, the file store.

One thing that hasn't been discussed (that I'm aware of) with regards to keeping it optional, and what was randomly mentioned in the other thread, is how modules can make use of the file store. In my opinion, these json files are by definition active configuration, not default configuration. Modules should be able to define their own default configuration, but they should do so in the code itself, not with json files. I like how the Variable module goes about this, where you implement hook_variables, and define the default values in one place, instead of being required to pass a default to variable_get().

I'm probably missing huge things in what you guys decided at DrupalCamp CO since I wasn't there to hear the conversations. :)

I lol'ed hard at the Twitter

te-brian's picture

I lol'ed hard at the Twitter joke :)

+1 for editable, active config files

mlncn's picture

Strong +1 for configuration files being editable, being expected to be edited- being the active store, not the database. It would be great if a module's default configuration came bundled with the module in the same format, so you could copy out the file or particular lines, and easily see what had been edited, with diff even.

benjamin, agaric

How about this

andremolnar's picture

How about this.

Write configuration in any structured file format.
XML
JSON
YAML
PHP (as a native drupal configuration object)

Have something that reads configuration files and converts them to native php configuration objects.
Cache those and bootstrap them.
If a configuration change happens - flush the cache - which rebuilds the configuration from the files.

andre

Caching PHP would require

Josh Benner's picture

Caching PHP would require (un)serialize, which is actually slower than json_(en|de)code.

Writing to different formats has a lot of inherent problems, including:
1. No native support for some desired formats,
2. Different formats represent complex data to varying degrees (ordering, special chars, etc)

Furthermore, I think having a pluggable system to deal with different formats would be nice, however core should only worry about a single out-of-the-box format. Then, in contrib we can make other formats work.

Maybe I wasn't clear

andremolnar's picture

By 'cache those' - just leave them accessible as executable php sitting in a file - and execute them. If configuration changes the file would need to be updated. No reason on earth to serialize them.

If I understand you, you are

Josh Benner's picture

If I understand you, you are proposing that we have Drupal write out files that will then be directly executed?

Caching PHP would require

Sylvain Lecoy's picture

Caching PHP would require (un)serialize, which is actually slower than json_(en|de)code.

According to this folk: http://groups.drupal.org/node/155559#comment-521629, I thought performance wasn't an issue.

In this thread: http://stackoverflow.com/questions/1306740/json-vs-serialized-array-in-d... They seems to says the serialize version is faster than json_encode. So maybe we'll need some real fact to judge about the speed of those functions. A link to some benchmark could allow people to tell if json_(en|de)code are really faster or not.

The debate around json_*()

pounard's picture

The debate around json_*() function being faster than (un)serialize is a wrong debate, it probably changes following PHP versions. Both are really quick, clearly, and you do not unserialize thousands of entries (by the way, all cached object are serialized, so JSON cached entries would be first unserialized, then parsed).

Pierre.

Going of in a tangent?

perusio's picture

The reason behind using JSON it's not because it seems to be favored by the majority. Rather the reason for going JSON is that it opens up a whole world of interoperability between drupal and everything else, IMHO.

Forcing PHP is a bad idea not only in security terms, but also because it forces someone else not privy to drupal internals to understand how the PHP associative array or object maps to their software. Consider for example Chef, being JSON is a plus and makes it simple to devise drupal related recipes. Also I think it could be a boon to a lot of third party contrib modules.

Actually, this doesn't sound

sun's picture

Actually, this doesn't sound like too bad of an idea at all to me. Why lock us down to one option if there are multiple possible ways to achieve a single thing? Update Manager/update.module in core along with its filetransfer classes is a comparable example.

Just define a clear and strict interface of how configuration data is expected to be converted. If you want, base that definition on JSON (as it's the most limited format and the protocol has highly varying implementations on different platforms).

The parser to use can be identified easily:

sites/default/config/foo.json
sites/default/config/bar.inc
sites/default/config/baz.yaml
sites/default/config/baz.xml
sites/default/config/baz.info

There's no reason for us to enforce a specific format, and your performance considerations may not have the same importance to someone else who's rather looking into interoperability with other systems.

Lastly, given a defined interface, writing those parsers is a matter of minutes.

Daniel F. Kudwien
netzstrategen

+1 Yes, this is probably the

Sylvain Lecoy's picture

+1 Yes, this is probably the best idea that i've seen here.

Do not lock people into JSON, just provides a configuration interface and let people provides their implementation for JSON, XML, PHP, whatever..!

We'd still have to pick one

webchick's picture

We'd still have to pick one format for all contrib modules and core to standardize on, in terms of how they expose their default configurations. I guess unless this was done via a hook implementation as q0rban suggested above.

We did at one point have a big long discussion about different formats, and I think it's worth breaking that out into its own separate "JSON vs. Alternatives" topic in the Build Systems & Change Management group so it can be adequately summarized and discussed on its own right, and not in a sub-thread of a sub-thread about something else entirely. So it'd be nice if we could table this discussion for now, and focus this thread more on the specific questions Greg posed above. We really need to figure out this security question regardless of what format we ultimately choose.

Bringing things back home...

webchick's picture

Just to re-orient the discussion back to the topic at hand... The situation we need to figure out how to deal with is the following:

  • One of the primary goals of this initiative is to allow configuration management in files (not just the database), so that they can be easily FTPed by beginner users, or can easily be managed in version control by advanced users.
  • These configuration files (regardless of what format they're eventually output in) will store values that include ultra-sensitive information, such as database passwords and whatnot.
  • The safest place to store these files is of course outside of webroot. However, not all web hosts provide access to directories outside of web root; for example, some shared hosts, or some hosts where you're not given proper access for some kind of security reason.
  • This means that in certain situations, you will be forced to store these configuration files in a web-accessible directory.
  • One way to protect files in a web-accessible directory is via .htaccess/web.config (or equivalent). However, these measures are not fool-proof. It's incredibly easy for someone's SFTP client to not show .htaccess and thus the upload gets missed. It's also easy for svn add to miss it. It's easy for your hosting provider to make a sudden upstream change that disallows .htaccess overrides. And so on.
  • And unfortunately, this is Drupal's problem to solve; we have an extremely large base of non-technical users who don't have control over their environment, and aren't going to blame CrappyHost when their site gets compromised by a GET request.
  • Therefore, we need some sort of secondary measure of protection in case .htaccess/web.config fails so the files are not readable. The old proposal was to embed JSON in a file called .json.php, because if a .php file was accessed directly, you'd just get a white screen (or in this case, a syntax error). Everyone hated it.

So, the question is, are the following two precautions enough?

  1. Make the automatic write-through to file-based config from the active store an optional, opt-in mechanism. So in the default case (non-technical user, shared host, etc.) they'd have only the UI/DB-based configuration mechanism, same as they do now, so no chance of someone reading their configuration files on disk, since they won't exist. Advanced users can choose to enable this option, and in so doing a hook_requirements() would prevent them from turning it on if a) the directory they specified was not outside of webroot, and b) .htaccess protection was failing.

  2. It's still possible for .htaccess to break even if an initial hook_requirements() check is good (.htaccess doesn't get moved to prod, etc.). So as a secondary precaution if this option is on, instead of storing files in a sites/default/config directory, put the files in a unique sites/default/config_8279s9f78sd9798as7d9as79d7sa9s directory, where 8279s9f78sd9798as7d9as79d7sa9s is a hash based on the site's private key. The idea is that this hash value would be internal only, and not something easily discovered by automated exploit tools. (Note: The directory only; the actual files themselves would just be prefix.name.ext.)

Does this work? What are we missing?

To answer specific questions

andremolnar's picture

"sensitive configuration information" has been in settings.php and the database forever and security concerns have been mitigated to the best of Drupal's ability up until now.

New security measures are always good, so no complaints on that front.

Can we do some detection during installation to decide what to do and how to do it? Not unlike clean url testing or other system probing and implement the best (most secure) option based on the environment?

No

chx's picture

No. I believe it was David who in the original thread have pointed out how even has misconfigured a server causing htaccess to fail. Relying on install time configuration is not the best we can do. I am all for always randomizing the directory name even for out-of-webroot configurations as properly configured permissions (aka read bit on the parent directory) can provide a quite some security even from local file system level attacks.

Of course, we can check .htaccess working and wave nice red flags if not. I wrote a sort-of-quine http://drupal.org/node/1183044 for this.

Well, and plus...

webchick's picture

Where you do your installation (localhost, dev site) is almost never where the site eventually ends up. So an install-time check is basically useless.

setttings.php is protected because it ends in .php. We tried to suggest ending .json files in .php and we saw how that turned out. :) So we're exploring alternatives.

Looking after non-technical users

mcjim's picture

setttings.php is protected because it ends in .php. We tried to suggest ending .json files in .php and we saw how that turned out. :) So we're exploring alternatives.

Yes, we should explore alternatives, but if ending .json files in php is the most practical, secure method for looking after non-technical users, then that might be what we have to do, regardless of DX.

I think we developers must be willing to compromise if that makes the whole system safer for everyone. We can update our IDEs, surely. I know I'm happier to have ugly + safe (even if I personally will be keeping my config outside of webroot. But this isn't about the likes of me).

Randomizing the name makes us

pounard's picture

Randomizing the name makes us either dependent of current config private key, or just dependent of "random". Scripting will be hard to achieve (I'm NOT speaking about Drush here).

If I use the configuration files over two Drupal instance (let's say my development box, and prod, to make a shortcut): because I don't use a VCS for database, I'll opt for the "rebuild/update" way instead of "dump the base" (which is the cleaner way): what will happen is the production environment will have either its own randomized configuration path (which is not the same because configuration cannot be loaded before configuration is loaded to get the right path), either is own private key. In both case, devel and prod env. will have different randomized dir, which make VCS usage impossible.

Once again, it's something that un-serve the original purpose.

Pierre.

Read

chx's picture

Read before you write. Thanks! (I already answered this scripting nonsense above, see.)

The config path can easily be made into a setting where the random dir is generated by install. And then you are done.

You are absolutely right,

pounard's picture

You are absolutely right, let's make it more obfuscated than it should be, complex solutions are always better than simple ones. I hope surely hope you will document using finnish language to ensure every one will be get to the point.

EDIT: Let me +1 the upper comment for the obfuscation beauty you made there.

Pierre.

Remember that private key is

webchick's picture

Remember that private key is per site, not per environment. The key.php file (or whatever it's called) would be uploaded to your production environment same as your index.php and sites/all/modules directory is. So both your dev and prod environments would come up with the same config_$hash.

This makes sens, thanks.

pounard's picture

This makes sense, thanks.

EDIT: So the key.php file will be easily discoverable then? Messing up a bit, but that's what popped into my mind.

Pierre.

I think the debate JSON vs.

Sylvain Lecoy's picture

I think the debate JSON vs. other format is actually the key. Its because you have chosen JSON that problems occur:

so that they can be easily FTPed by beginner users, or can easily be managed in version control by advanced users.

default.settings.php is the perfect example.

These configuration files will store values that include ultra-sensitive information

settings.php is doing it great.

The safest place to store these files is of course outside of webroot.

Not anymore with the .php extension.

One way to protect files in a web-accessible directory is via .htaccess

Its build in for .php extensions, no need to configure !

we have an extremely large base of non-technical users who don't have control over their environment, and aren't going to blame CrappyHost when their site gets compromised by a GET request.

This is JSON related problem, actually with php extension you can't GET it. By the way, because of that we are creating a randomized folder to obfuscate the configurations files. This is not needed for .php (all knows that settings is located in sites/default).

to embed JSON in a file called .json.php

This is again a problem introduced by the JSON format.

To people who will tell me "Don't feed the troll" think twice please, think that its not at all a troll, all the technical problems come from the hypothesis of using JSON as format. All the problems mentioned above exists because JSON introduced them. So this is completely in-topic.

I very much am in favor of

Lars Toomre's picture

I very much am in favor of having a configuration API that can read in configuration data from whatever structured format (.json, .xml, .info, etc) and convert that information into whatever Drupal format needs internally and supports by default. Ideally, this API would also allow one to write the internal data configuration out to disk in any of these formats (or even to a config SQL table for those sites that need to manage config via databases).

Clearly, the most secure way of managing file-based config is not to have the config files exist at all. Hence, I am a big fan of having this be an opt-in feature that one must enable to get all of the goodies that come with file-based config. Presumably, there can be some inefficient fallback like exists now. Also, I assume that there still will be something akin to settings.php which can include values for both whether to use file-based config and what directory (ideally external to webroot) to find those files in, and the private key.

The second suggestion for a hashed config directory if in the webroot is a good one. I am not sure if that needs to be the case also if placed outside the webroot.

I also believe a third level is required if the files are being written within the webroot. One never can predict how these config files might be exposed. Hence, I think that it is extremely important that sensitive information never be exposed, even if it contributes to developer WTF. Hence, if the config files need to be stored in webroot somewhere, I strongly feel that they need to be stored there with *.php extensions and the die() code. If stored outside of webroot, they can be written in pure JSON or whatever people decide makes for the most efficient performance.

One other note about config files... If the files are human editable, I am aware of several clients that will need to have a comment facility within those files explaining what this or that configuration value means or why a setting was changed to such and such. Does this eliminate JSON format for such clients because it does not support comments?

Configuration

andremolnar's picture

Allow me to make the following assumptions we hopefully all agree on.

  1. Configuration is: Data containing the current assumptions the application will use to run.
  2. The fastest way the application can read/understand its configuration is if it is in a native format to begin with (in this case PHP)
  3. The slowest way the application can read/understand its configuration is by querying a database in a non native format and having to parse result into a useable format.
  4. Other methods of configuration retrieval fall somewhere in between
  5. The application rarely ever (never) needs to know every possible configuration during every execution
  6. Configuration should be accessed by the application as needed (lazy loaded)

Let me make the following observations about configuration storage.

  1. Database configuration storage is easy: to access; to write to; to persist
  2. Database configuration storage is hard to: deploy; version control
  3. Configuration in structured data stored in a file is easy: to access; to write to; to manipulate (by hand); to deploy; to version control
  4. Configuration stored in native PHP code is easy: to access; to deploy; to version control
  5. Configuration stored in native PHP code is marginally harder to: write to; manipulate by hand,

Given all these assumptions and observations I would argue that:

An ideal configuration system would generate PHP code (configuration objects) that is written to disk.
During bootstrap and as needed during application execution these objects can be loaded/instantiated.

What's good about this? Its fast and configuration lives in code.
What's bad about this? Nothing.
What's tricky about this? Security (a solvable problem). Code to generate code (we do it all the time anyway).

But, what about module developers and site administrators? Are they going to have to crack open an IDE just to make configuration changes?

If they want, yes go ahead and edit PHP, but that's not realistic in all cases.
Developers may choose to write configuration files in any number of supported* formats. For example JSON, XML, YAML.
(*supported format being any structured format that conforms to whatever rules we decide is considered valid)
Site administrators will have Drupal between them and the configuration, so they don't have to open any files if they don't want to.

But, what is Drupal going to do? Constantly read and parse configuration files? Constantly re-write php classes on the fly? That's going to chew up my CPU and bog my system in disk IO.

Not if we're smart. A number of solutions are available but I imagine the following:
Non PHP configuration files could be parsed and cached (in the database or memory) as an intermediate step.
Then on demand, via a click of a mouse or drush command, the configuration would generate PHP code that is written to disk.

But what about those userland configuration changes that you said Drupal was going to take care of?

Right. if a site user makes a configuration change on screen - then Drupal writes to whatever format it wants initially AS LONG AS the data is soon thereafter exported to any supported structured format AND its available to be converted to a PHP configuration object on disk.

So why did I write all this?

To explain in gory detail why it makes no difference whatsoever what format configuration is stored in INITIALLY as long as at the end of the day it ends up stored in some sort of native PHP object that is loaded by Drupal.
Module developers and Chefs and everyone else can store configuration in whatever format they like to work with. Drupal can pick whatever format is likes best, and use that when converting userland clicks into files (or database entries).
Since all the formats have to have rules in common, configuration can be easily converted from one format to the other, and all can be converted into code.

I hope these initial thoughts make sense. I don't think what I'm saying is a departure from the thought that has gone into this already, but offers some new thoughts on how it might be implemented.

andre

Posts passing in the night

andremolnar's picture

My post was being composed as webchick 're-oriented' the conversation. Sorry if I end up derailing the conversation again.

No I liked it, at least it

pounard's picture

No I liked it, at least it definitely get rid of the format performance debate.
I really like PHP configuration, at least it would get rid of the randomized dirname nonsense.

PHP configuration is much easier to read and write than JSON, definitely, and allow commenting inside, just as the actual settings.php file. Instead of arguing the choice of this or this format, it would be a non-regressed and natural evolution of the actual settings.php file until some other solution that makes everyone happy has been found (and it causes less security problems, at least these won't be readable from the outside).

The original post that came up with all of this long running threads is only a 3 hours sprint, a lot of energy spent on this. Using good old PHP files for conf is probably the only way to stop it and start to focus on what will be the actual API for using the configuration, which is IMHO a much more important point right now because configuration initiative is blocking some other initiatives. Storage is just backend, not API. API is needed for the others while they really don't care about storage is the API is good enough to mask it. And right now, the proposal API's is directly derivating from the storage definition (this and the followed up thread), which is definitely an error because it reduces really nice API potential solutions range, that could definitely be a lot more high level without having that much impact on performances.

Pierre.

Correction

gdd's picture

The original post that came up with all of this long running threads is only a 3 hours sprint, a lot of energy spent on this.

Actually the original sprint was three days and it ended up spilling into 4th specifically because of this file format issue.

PHP performance

catch's picture

Since we're talking about default objects and the level 1 storage here, both of which may need to be read at once for a full sync, I did a quick test between including a PHP file defining an array, and getting the contents of a json file and decoding it, posted the results at http://drupal.org/node/1198924

DB credentials

Crell's picture

My only concern here is that some config must live on disk, because it's needed to get to whatever other datastore we're dealing with. Specifically that's things like the DB credentials, memcache info, Redis configuration, etc. Of course, that also happens to be the most sensitive information we have.

So does that force that high-sensitivity stuff back into a for-reals .php file like settings.php? I'd hate to do that, but is that what we'd have to do?

What exactly is so bad about

webchick's picture

What exactly is so bad about this? As catch said in the other thread somewhere, the settings.php/conf.php/whatever pattern is very common among PHP applications, so it's not like this is a weird Drupalism or anything. And it's worked fine for 11 years. :)

Yes, totally agree with you

pounard's picture

Yes, totally agree with you on this.

Pierre.

A few reasons

Crell's picture

1) Several people were hoping to be able to eliminate settings.php entirely. I admit I didn't think it would be possible, but several people were hoping to.

2) It's a totally different format (straight PHP) from the primary config system (JSON). It may or may not map well. That will be an implementation question.

If that's what we end up doing I am not going to make a fuss. I'm just observing that it forces us to have a .php config file. We then also need to consider if we flag that file as always-local (ie, settings.local.php becomes settings.php, essentially) or if we also need a separate local-override mechanism, and it's the same or different from the JSON override system.

I think that an override

pounard's picture

I think that an override system will be nice here. For example, some environment related plugins low level configuration (early bootstrap cache backend is a good example) might be in there. From a development workflow perspective, I could totally develop with APC cache for early bootstrap caches on my development box, and have it all on a memcache on production environment. This simple fact needs the settings.php to have a "no conflict/override" mecanism for versionning. Actual solutions with stable Drupal can be, for example: not pushing the settings.php on the VCS but a series of differents settings-.php and link the right one on the right site. It works fine, but it could be even better.

Pierre.

: )

itabspst's picture

I loled hard at the twitter joke, too ;)

Another followup

gdd's picture

OK so what I read from this thread is that, if we assume JSON, then everyone is pretty much OK with the suggestion of a) using pure JSON b) in a hashed directory name for storage c) with storage turned off by default. However there IS controversy around the choice of JSON at all. Therefore I have opened yet another discussion to hash out the format issues. Please head over to http://groups.drupal.org/node/159044 for that discussion. Thanks!

We could keep config in the

seandunaway's picture

We could keep config in the database, the way its always been done, with one modification...

Give the option to keep 'config' tables in a separate database than the 'content' tables.

mysite
mysite_config

Then I can just mysqldump (or drush export) the config db and version control, deploy, etc.

For 98% of Drupal users, nothing changes. It's fast, its easy, it's secure, it's familiar.

It seems most of the issues are trying to figure out how to write these out to the file system with the associated security / performance implications - is that even necessary?