Features module quick demo

yhahn's picture

Jeff Miccolis and I spent some time last week pulling together a proof of concept -- we're calling it the "features" module. Jeff writes:

/**
* @file
* Module file for the features module, which enables the capture and
* management of features in Drupal. A feature is a collection of Drupal
* entities which taken together statisfy a certain use-case.
*/

Here is a quick informal screencast of what this thing does from the UI perspective:

http://video.devseed.s3.amazonaws.com/features_mar13_2009_p1.swf
http://video.devseed.s3.amazonaws.com/features_mar13_2009_p2.swf

What will be of more interest probably to people in the group is the internals of the module. If you're interested, take a look at README.txt and API.txt, where I've documented some of the basic concepts for feature component and module dependency detection. The core concept behind features is that you can start from some starting point in Drupal and allow each derivative component to delegate export and dependency detection to its derivatives and so on.

From API.txt:

In practice, hook_features_export() has 3 tasks:

1.  Determine module dependencies for any of the components passed to it
    e.g. the views implementation iterates over each views' handlers and
    plugins to determine which modules need to be added as dependencies.

2.  Correctly add components to the export array. In general this is usually
    adding all of the items in $data to $export['items']['my_key'], but
    can become more complicated if components are shared between features
    or modules.

3.  Delegating further detection and export tasks to related or derivative
    components.

#3 is a key concept in the Features module. Each export processor can
kickoff further export processors by returning a keyed array where the
key is the next export processor hook to call and value is an array to be
passed to that processor's $data argument. This allows an export process to
start rather simply, at a single object:

  [context]

And then branch out, delegating each level of branching to its
appropriate hook:

  [context]--------+------------+
      |            |            |
    [node]      [block]      [views]
      |
    [CCK]
      |
[imagecache]

I'll be writing up another post after this describing some issues we ran into, in particular with the state of exportables in modules.

Comments

bhuga-gdo's picture

This is pretty awesome. Much more well thought out than I would have designed, and I love that it works with context. For brevity's sake, I will forgo the well-deserved adulation that this deserves and move on to possible issues, as saying how awesome this is would take longer than I have to write. And please don't take this as a 'this can't work' post--it's meant as a question and discussion, not a 'this sucks'.

Question: what do you do with modules that have been altered? For example, let's take the system module, and in particular, the performance page. The very popular boost module modifies it; it actually replaces a fieldset-ful of elements with its own.

<?php
   
case 'system_performance_settings':
     
module_load_include('inc', 'boost', 'boost.admin');
     
$form['page_cache'] = boost_admin_settings($form['page_cache']);
     
$form['#validate'][] = 'boost_admin_settings_validate';
     
$form['#submit'][] = 'boost_admin_settings_submit';
     
$form['clear_cache']['clear']['#submit'][0] = 'boost_admin_clear_cache_submit';
      break;
?>

Leading eventually to this:

<?php
function boost_admin_settings_submit($form, &$form_state) {
 
variable_del('boost_previously');
 
extract($form_state['values'], EXTR_SKIP | EXTR_REFS);

 
// Forcibly disable Drupal's built-in SQL caching to prevent any conflicts of interest:
 
variable_set('cache', CACHE_DISABLED);

  if (empty(
$boost) && !empty($boost_previously)) { // the cache was previously enabled
   
if (boost_cache_clear_all()) {
     
drupal_set_message(t('Static page cache cleared.'));
    }  
  }
}
?>

Note the variable_set on Drupal's built-in cache. Since boost has taken over the form with form_alter, there's no other way to talk to the cache module.

The test cases are:

  • system.module implements features_export_render but boost does not. Here, system module will probably not export the page cache as enabled, because boost has disabled it. But is that correct? Boost is essentially preventing this module from exporting its settings.
  • Boost implements features_export_render but system.module does not. Here, Boost has its work cut out for it. It's got to re-implement most of its settings because they currently exist inside form API calls, which is generally the case for most modules, especially those that alter other modules.
  • Both modules implement features_export_render. Here's the real twist. The onus should not be on system.module to know about boost, and it has things unrelated to caching to export. Meanwhile, boost is trying to overwrite particular system.module settings. At the very minimum, system.module will export semantically extraneous data about caching methodologies that are not being used, and are decidedly unrelated to the feature module being exported, which one assumes is using boost.

In any case, editing the resulting feature module can get you into a state that has been more or less declared as undefined by the boost module.

I've picked Boost as an example because I've done some coding on it and I happen to know how it works offhand. I'm not really sure how often this kind of overlap happens with config--a few minutes browsing through some popular modules reveals that most form_alters seem to alter content forms, not configuration forms, which may place my concerns outside of the original scope of the BoF and API. But I think that this kind of issue means that there could be a lot of trouble trying to expand this to even basic content. We knew this was going to be a gray area, but with so many modules adding information to node forms, can this solution work for, say, OG nodes? Is this not even common enough to worry about?

This is going to be a question for any solution that is not built on top of Forms API.

Would this maybe be solved by a module that maps form API calls to some kind of config? That is to say, one module that implements these hooks and then provides other, simpler hooks to other modules, allowing them to export some settings that end up coming back to them as form calls. Implementing boost's settings in this way would let boost and system.module avoid clobbering each other semantically, and would let the maintainers of individual modules decide if they risk this kind of semantic collision and how they wanted to implement things.

If the Configuration Framework being worked on by the Gravitek folks were to implement this hook, is that more or less what we would have, or am I misunderstanding something?

Also, one other, small question: I suppose I'm resigned to the idea that exported config needs to be code, much as I would like for our QA people to be able to edit it. Is there a precedent in place for a separate repository of modules of a certain type? The blog feature module and the blog module might share the same namespace in code but they need not necessarily be in the same drupal.org module system.

Ben, your post pretty much

yhahn's picture

Ben, your post pretty much touches on the most difficult questions at this point -- what to do with variables, and what to do with API's based around form_alter().

You'll notice that what we are exporting at the moment are things that the implementing modules already have frameworks for exporting (node types through hook_node_type_info, views through hook_views_default_views, etc.). This keeps our module simple but it also means the scope of the module stops where the scope of the implementing modules stop. It should be a good motivator for getting solid exportables frameworks into modules and also thinking twice about using a variable for configuration when it could be more robustly represented in your data structure. For example, imagine the havoc that would be caused if Views used variables to store actual information about the view, or if you used form_alter()'s to implement your extensions to Views rather than the built in handlers/plugins architecture. Not pretty.

So the short answer to your questions are: amen, these are the hard problems. Right now there are no plans to tackle these head on in Features -- I think the Patterns guys are taking a lot of steps to try to tackle form-based configuration storage as it exists now.

To frame the problems better for future discussion:

  • Variables are currently a real conundrum to export and manage cleanly. First off, there is no way to know all the variables that a module provides. Some, like the node type variables, practically exist out of convention ($form_key .'_'. $node_type). One possible first step is the hook_variables() patch which would require you to declare the variables you're using. http://drupal.org/node/145164
  • Forms in your API cause additional problems because they don't let you extend someone else'd data structure -- rather, you tend to hijack their submission process and store what you need, in your own variables usually. This makes it hard to export the "settings" for this form, or even have some definite concept of where one module's settings stop and another's starts.

Some thoughts on variables

nedjo's picture

I posted a relevant issue here http://drupal.org/node/551736.

This !!Rocks!! and is pretty

ChrisBryant's picture

This !!Rocks!! and is pretty much the other side of what we've planned to do with Patterns/Configuration Framework (primarily the use of available import/export apis, creation and exporting front end, as well as Drush integration.) The UI is great and I love using context to draw a circle around a feature and then export it whole. I assume at some point this might not require context and have a similar interface for choosing which Drupal entities you want to add to the feature.

You mention that the planned scope of the Features module ends at modules that don't or won't implement exportables. Does that mean that you aren't open to an approach that uses a fall back method to FormsAPI when proper importables/exportables APIs aren't available. I understand that module developers need to be given an incentive to use exportables but if that's the only way that's supported then that will limit people from having a comprehensive solution for a while until those hooks are in Drupal core and every module is required to use them. That means we would likely be looking at Drupal 8 some 2 years or so away.

Also, what if a module's API doesn't support everything you can do with the module? Take for instance module settings vs. config vs. data in the Aggregator module. You have settings settings such as when to discard feed items vs. the feeds themselves. Aggregator would need to support exportables that included settings separate from the source feeds you add, but maybe those don't make sense to be part of the exportables?

It would be great if we can work out a way to combine forces and support both methods. Then we can give everyone something that can be used on Drupal 6 and onward. This could be accomplished through the appropriate hooks and separate modules or by integrating the functionality into one module. If not, then it means more modules and choices in the community doing the same or similar things with overlap and duplication of efforts.

I hope we can arrive at a comprehensive solution that will work for Drupal 6 and 7, but I fully understand your desire and need to limit the scope and provide incentive for module developers to change their ways.

--
Gravitek Labs

I think that CF / Patterns can be the glue here

bhuga-gdo's picture

I think the answer to the Aggregator question is that aggregator decides. features_export does not require you to export every view, you pick. Right now you pick them via context, but I don't see that as integral to how the module works, just a good starting point. Aggregator should be able to let you export a few things--'global settings', 'feeds', whatever people want it to work into it to support. I think that this is a smaller problem than we think--I think that most modules have a clearer separation between settings and content. But when 'content' can be considered a 'feature', aggregator can export its content--even if its as a patterns macro.

I think that Patterns can be used as a source here for the features module. You don't get the nice override functionality, but you get something. Here's an example roadmap:

d6

Patterns can be added as a source in the features module. This might be more appropriate as the Configuration Framework, but I'm really not familiar enough with it to know. I'd really like to have one of you guys comment on this specifically. This gives us a pretty simple interface to a lot of modules that are not realistically going to be updated to support this kind of thing before d7.

The end result is a set of first and second class citizens for configuration management: those that can be managed with patterns, in a recorded-macro fashion, and those that can be managed directly, in a declarative fashion. The resulting modules can run patterns on install, perhaps. The community decides what's worth implementing.

The various competing use cases can reuse this hook how they want. I think that features needs to be careful to separate the use of the hook from its UI, but if does, there's no reason that the UI needs to be this--it could also be a something that automatically compares a feature module in the background with a running copy, for example, resulting in solutions for a lot of common use cases.

d7

We have the same situation in D7, but if the variables declaration patch gets in, then most simple modules can have their config generalized into a declarative fashion by some module (probably a part of features, or whatever features becomes).

At this point, we'll have almost solved the problem of developer best practices. Modules that have config so weird that it would still require a pattern (and there will be some!) will be decidedly second-class citizens compared to those that are simple enough to be managed with declared variables or by implementing the features export. Of course, it will all come down to how much the community wants it, but it always does.

I imagine some modules will still have edge cases that need patterns--aggregator might still be an example.

d8

Probably something completely different.

But is it core?

So features is pretty awesome, and code walks while gdo talks. But we should ask if this is the kind of hook we want in core for d7.

The common use cases from the BoFs were:

  • Dev / Staging / Production dev cycle
  • Enabling / Disabling a swath of functionality all at once
  • Automating away common tasks
  • Updating complex configuration on multiple sites (almost always views / cck)
  • Dependency resolution

I think that this hook actually supports these quite well if supported by patterns. Some things will never really be proper for a feature module--for example, actions with side effects, such as creating users. For example, a simple dev / staging / production cycle can be made with most of the fun stuff being in feature modules, but with supporting patterns for adding content and tweaking fringe modules. You can't update content, of course, but we more or less agreed that content management was outside the scope of this project.

One problem I see with the hook as designed is that modules can't implement parts of the hook for other modules. This may not be a problem, but I can imagine an instance (aggregator again) where the main module might implement a the hook for settings, but someone else wants it to implement aggregator content as an exportable as well. As implemented, this will lead to a hack-core situation. We'd need some kind of hook where modules can register 'hey, I can export stuff for aggregator too. Here's a callback to run when you ask aggregator for what it can export that will add to that list'. Note that I would advise explicitly against features_alter, or we'd just end up at the problem with Forms API again. Something like features_add.

Packaging & Deployment

Group organizers

Group notifications

This group offers an RSS feed. Or subscribe to these personalized, sitewide feeds:

Hot content this week