The Core Initiative Trinity

We encourage users to post events happening in the community to the community events group on https://www.drupal.org.
Crell's picture

The twin deep-engineering Core Initiatives of Configuration and Web Services have been taking a long hard look at Drupal's basic, underlying architecture of late. We're swirling around three fundamental pieces that should, we believe, revolutionize Drupal core and Drupal development, allowing for a quantum leap in Drupal's capabilities. (Excited yet?) The catch is, we have to get these right. More than that, we need to nail these. For that reason, we're trying to solicit wider feedback on a few key architectural questions. I have tried to summarize the situation and the challenges below, and invite informed, insightful feedback from the community.

The three cornerstones of this new approach are Configuration, Context, and Plugins. All three are things that Drupal core does not, in fact, provide in anything resembling a useful way. We need to get all three of them into core, however, and they need to work together well. The problem is that there are very good reasons for all of them to require one of the others first, which creates a circular dependency. We need to sort out that problem before we can get building.

Configuration

At present, Drupal core offers only one configuration system: variable_get(). That is, the only configuration system available in core is a single, inefficient, unsearchable key/value store. All other forms of configuration in Drupal, even in core itself, are one-off implementations, most of which use the database directly because, well, it's there. That has led to the current situation where managing configuration in anything resembling a sane fashion is ugly, a hack, or both. "Exportables" are currently a widely popular solution, but are still backward because it puts the database as the primary source of information and we still have no standardization whatsoever for those systems.

The Configuration and Deployment initiative, headed by Greg Dunlap, is looking to solve this issue by replacing variable_get() with a more robust and extensible in-core configuration system that stores data primarily on disk, which makes it a hundred times easier to version control, deploy, and manage. And by offering more features than variable_get(), we can get rid of most or all of our one-off configuration systems and fold them down into a single, powerful, deployable system. By basing it on the file system, we can make it available at an extremely low level very early in the request cycle.

For more information, see Greg's battle plan writeup.

Context

Drupal core currently offers almost no mechanism for tracking or handling context. By context, we mean useful information that makes one request different from another; the requested URL, the objects specified by that URL, the current user, the current language, the type of data the user agent (usually the web browser) wants back, what section of the site we're in, etc. Every single one of those pieces of information may be extremely important, but every single one of them is currently handled in a completely different way. The closest we come to handling context is the menu load arguments passed to menu callbacks, which derive from the URL. For pretty much everything else you can think of, you're totally on your own. And don't even think about trying to properly unit test that.

The Web Services and Context initiative, headed by Larry Garfield, aims to introduce a unified context broker that can be used system-wide. By replacing a hundred one-off utility functions that don't talk to each other and are all simply a wrapper around global state with a clean, unified system for taking requests for contextual information and mapping those to the proper code to determine that information, we can streamline the way Drupal responds to incoming data. It also will allow us to more cleanly segregate parts of the system while still allowing them to share important information, which simultaneously gets us better unit testing and makes it easier for different systems to integrate with each other.

For more information, see the WSCCI definitions page.

Plugins

Drupal core currently has no system for plugins at all. Plugins are not hooks; hooks are event handlers, in essence. Plugins are configurable, swappable implementations of some task. That could be alternate cache backends (currently classes with a common interface and hard-coded $conf settings); it could be alternate session storage (currently swappable include files and hard-coded $conf settings); it could be blocks (currently a one-off SQL table that no one really likes); it could be Views (currently a named, nested set of custom objects and arrays with a custom serializer); it could be text formats (currently an info hook that then gets stored in a totally custom way); it could be Field widgets (currently pseudo-hook callbacks that are architecturally backward); it could be the type of HTTP response to send (currently not pluggable at all, which is a problem). Every single one of those is its own one-off implementation. And most of them suck.

The Web Services and Context initiative also aims to introduce a single, in-core, robust plugin system to unify all of those systems and more. By having a single, robust system we can vastly reduce the amount of code in core, we can vastly reduce the complexity of core making it harder to introduce bugs, we can vastly simplify core's API so that there is only a single system for new developers to learn, and we can make pluggable things that currently are not. That opens the doors for fascinating concepts like swapping out implementations of core logic, moving Drupal to an entirely non-SQL environment, substantially more robust handling of web services and REST requests, and other fun things.

Note: Plugins are not intended to replace hooks; they complement them by solving a problem that core doesn't currently solve and for which hooks are an extremely poor fit.

For more information, see the WSCCI definitions page.

The problem

Those three key changes are fundamental to Drupal, but are critically important. So what's the problem?

* The problem is how those three pillars relate.
* We want the configuration system to be flexible, and allow for more than just one hard-coded implementation. That means it needs plugins.
* We need the plugin system to allow plugins to be configurable. That means it needs configuration.
* We need the plugin system to be able to respond based on more robust request information. That means it needs context.
* We need the context system to support swapping out what code is responsible for handling what piece of contextual information. That means it needs those handlers to be pluggable.
* We need to be able to configure what code is responsible for handling what piece of contextual information. That is, it needs configuration.

That creates a horrible mess of circular dependencies, and we're dead before we even start. We have to resolve the relationship between these three systems before we can put pen to paper (or finger to keyboard, as it were).

Possible solutions

A number of possible ways around this problem have been bandied about, but not yet settled on. I have attempted to document the main ones below. (Note that these are not mutually exclusive.)

1. Don't make configuration storage pluggable. Just accept that there will be one (1) way that configuration gets stored and be done with it. Possibly we could allow only certain parts of configuration to be pluggable, those that are not needed early on.
* Pros: Eliminates the dependency on plugins, simplifies the implementation, and ensures that configuration is the baseline system upon which everything can be built.
* Cons: Little to no flexibility. If you have multiple web heads then making changes via files on disk are actually harder to propagate than database changes.

2. Implement a simpler, more streamlined pluggable mechanism for context handlers other than plugins.
* Pros: Eliminates the dependency on plugins. Allows for a finely-tailored light-weight system in a very performance-sensitive area.
* Cons: We're back to having multiple pluggable mechanisms, even if we only plan on the one one-off.

3. Implement a simpler, more streamlined pluggable mechanism for configuration other than plugins.
* Pros: Eliminates the dependency on plugins. Allows for a finely-tailored light-weight system in a very performance-sensitive area.
* Cons: We're back to having multiple pluggable mechanisms, even if we only plan on the one one-off.

4. Allow context to be optional for plugins, based on their type.
* Pros: Plugin types that don't need context don't have to deal with them. We can use plugins for context handlers because we then don't have a circular dependency, so there's more consistency.
* Cons: Adds complexity to the plugins system, since there are multiple code paths. You may define a plugin type to not need context, but then oops, one particular implementation may need it and then you need to try and hack your way out of that hole.

5. Allow configuration to be optional for plugins, based on their type.
* Pros: Plugin types that don't need configuration don't have to deal with them. We can use plugins for configuration, because then we don't have a circular dependency, which allows configuration itself to be swappable.
* Cons: Adds complexity to the plugins system, since there are multiple code paths. You may define a plugin type to not need configuration but then oops, one particular implementation may need it. (We saw this with field formatters in Drupal 6.) Still doesn't solve the problem of how you configure which plugin to use for configuration.

And probably others. We need to settle on some combination of these trade-offs in order to properly implement these three key systems.

Proposals

There are two main proposed approaches so far, which I will try to adequately summarize.

In proposal A, we basically do #1 and #2 above (or possibly #2 and #3). That is, configuration is the bottom of the stack. It depends on nothing except having a filesystem that it can access (either via fopen() or SplFile) and the PHP JSON libraries (which have been standard on PHP installs for years now). We then use configuration and a custom one-off system to build our context handling system. We then use configuration plus the context system to build a plugin system. Plugins then always take context and configuration, consistently. If a given plugin implementation has no need for one or the other it simply ignores it, but it's there just in case. If we decide that configuration handling needs to be pluggable, we do that with a custom, low-level approach as well.

In proposal B, we do #4 and #5. Plugins, then, become the fundamental underlying layer. We eat the extra complexity of a single plugin system with multiple usage modes, and then use plugins in "reduced mode" to power both swappable configuration storage systems and the mapping for context handlers. Configuring those early plugin types is done via custom PHP code.

And there may be other combinations we're not fully exploring yet, but need to.

Note that all of this work is happening very early in the bootstrap process, so it must all be fast and Drupal's usual hammer, "when in doubt add a hook", is not available to us at all.

RFC

At this point, we'd like the thoughts of the rest of the community on how to move forward. Every option has significant trade-offs, and we need to decide which trade-offs we want. This may be one of the most important decisions we collectively made for Drupal 8.

Please keep comments on topic and constructive. :-) Let the bikeshedding commence.

Comments

Now editorializing...

Crell's picture

My own view is that we want the main plugin system to be as consistent as possible. Generally I support approach A above.

1) The mapping logic inside of the context object is very specific. It probably will not benefit from the more generic, generalized mapper available to plugins in general. Also, it is really really important that the context system be fast. That means removing abstraction layers internally. If all we rely on for making context work is the config system and a few arrays, that's good.

2) We are planning to eliminate all of the various one-off hacks for context. global $language, global $user, arg(), menu_load_object(), etc. should all be dying. If you decide you need that information in a plugin that doesn't have context, those old cruches will no longer be there. While we are planning to still have a procedural way to access the top-most context object for legacy code, the intent is to avoid that as much as possible. It also makes unit testing harder since you cannot always just directly inject the context object. We would also need to move, probably, from constructor injection to setter injection, which is admittedly not the end of the world but is not as reliable as constructor injection.

Basically I want to keep the number of possible conditions that a module developer writing a plugin has to think about to a bare minimum. I'd rather pass around empty objects that behave in a predictable manner than sometimes have them and sometimes not. I see it as the same concept as using array() rather than NULL to indicate "no elements to process", because then you can safely foreach() over it the same way either way.

As a primary proponent of

merlinofchaos's picture

As a primary proponent of option B, I think the 'multiple code paths' issue is somewhat overstated. IMO, we need multiple code paths to be able to deal with early system where things like databases or users or the like are not yet available. That means enforced dependencies are bad. And yes, we can have a plugin system that has all those dependencies, but IMO it just means that those systems that need one-offs have systems that do substantially the same thing, but with a different code path.

Which means you have the same issue just organized differently.

Now, my ideas are that the basic settings when defining a plugin type give you what the majority case will want. It's difficult to talk about this without getting into code specifics, but the summary is that majority case wants to end up with an instantiated plugin object that you can call methods on to do things. The difficulty is configuration, and experience and research indicates to me that where configuration comes from cannot and should not be specified by the plugin system. In a view, the configuration for that plugin is somewhere on that view. In imagecache, the configuration for that plugin is in the derivative. In the session plugin, configuration is probably in settings.php or possibly a subsidiary file because we have very little we can rely on at that point.

The main issue there is that some plugins have more or less global configuration, some have more or less local configuration, and the plugin system will do us a disservice by constraining how that configuration gets there. As long as it provides patterns for the most common cases that are easy to use, then we'll have a good system.

I also think it's important

merlinofchaos's picture

I also think it's important that some plugins just aren't objects at all. There are many uses for theme-y plugins where it's really just a template or set of templates. i.e, a panel layout or an advanced forum style. There is little need for an object here and requiring that the plugin have this object makes no sense to me.

Succinctly, from my

merlinofchaos's picture

Succinctly, from my perspective, the conundrum boils down to this:

Most plugins need to be configured. Does that fact make configuration a requirement of the design of plugins?

Or do we allow the possibility of unconfigurable plugins, so that the configuration system can utilize plugins to swap out configuration storage? The con of this seems to be that if plugins don't HAVE to have configuration, it will be confusing and inconsistent to developers. I don't actually understand the basis of this part of the argument.

I fully support that plugins do not need context or configuration, and it is desirable that configuration be pluggable. It therefore makes the most sense that configuration depends on plugins, and not the other way around.

Sounding Off

EclipseGc's picture

Also as a proponent of B, I'm going to highlight something merlin said (I think) in a different manner.

Forget option 1, and consider implementing just options 2 and 3 for a moment (for the sake of argument). If we do this, then we've fundamentally done 4 & 5, the only difference here is that the only plugins that aren't required to consume both context and configuration are the one off systems we build for them... all other plugin types are shoehorned into a solution they may or may not need. Years of working with ctools plugins now have taught me that every plugin is different. The primary plugin type I work with and create on a daily basis is administrative in nature, and thus doesn't care about context at all (in the broader sense it cares about path, but the menu system handles that which is a completely separate can of worms from this discussion). Likewise the plugins are primarily concerned with local configuration. Contrast this with a plugin that can determine page accessibility, and you ABSOLUTELY need context and configuration in virtually every case.

Anyway, my point is really that doing 4 & 5 doesn't imply a plugin system that's fragmented and confusing, and that ultimately NOT doing 4&5 (and doing 2&3 alternately) really just cripples the potential extensibility of both config and context by not maintaining them on the same plugin framework.

Bottom line, the only systems that suffer from plugins NOT coming first are the config and context systems. Plugins, as the 4&5 supporters think of them, have use cases both with and without context/config... that ideal doesn't preclude consuming context and configuration though.

Eclipse

While having the

pounard's picture

While having the configuration in the design, you can still use stateless objects that doesn't use it, with no cost at all over performances. Don't forget that object's methods are basically shared function pointers in memory, and that having 10 methods on a class won't perturb a bit the execution time if you don't use them.

Pierre.

That would imply building

neclimdul's picture

That would imply building context and config into the design and not having a requirement on it, not having a base requirement for it.

And why not? If Crell has

pounard's picture

And why not? If Crell has posted this post on the two groups, it's not for nothing.

EDIT: Configuration layer is really an ultra-low level layer. And light as proposed. It should definitely be the common base. If it doesn't, then IMHO this new plugin system would only be a pure CTools port in OOP (or maybe not nothing is fixed anyway) which is IMHO a huge error.

CTools for what it worth (and it has good concepts in it, and really if I doubted that before, after a long talk with EclipseGc I'm sure they are good things in it), is now aged and was designed on totally different paradigms (pure procedural code, lack of core low level API and helpers, etc). Core is changing, it's time to merge, but not only merge, it's time to redesign, not port.

You know there is a theory in computer science, is that V1 of a piece of software is always crap, that V2 is stable version of crap, and V3 is redesigned software that learns from the errors of the past. This theory states that every software only gets stable and usable in V3. I couldn't agree more, while this not about version numbers, this is also about concepts and design. I think that today, CTools is still V2, stable version of crap, even if it carries really good concepts (and please, Earl, this is not personal, and this not about bad coding, this is about history and aging) it needs this redesign.

Pierre.

Simplified

EclipseGc's picture

Conceptually, the whole version argument aside, that's exactly what's being done. Neclimdul has put an enormous amount of work into rebuilding the plugin architecture in OOP and removing as much of the cruft as possible to make it pure awesome. But that's not really related to the discussion at hand. The "ctools" people are saying "Plugins don't need configuration and context, if they have it, awesome, but they don't NEED it." If we accept that statement as "fact" then where does that leave configuration and context? It seems to me that if plugins are independent optional consumers of both configuration and context, then context and configuration can be dependent upon plugins. That is essentially the crux of the argument, and the only thing you can really say to debunk that is to deny what the "ctools" people are saying.

Therefore:

<?php
$truth
= "Plugins don't need config/context";
if (
$truth) {
 
$config->can_use_plugins(TRUE);
 
$context->can_use_plugins(TRUE);
}
else {
 
$config->can_use_plugins(FALSE);
 
$context->can_use_plugins(FALSE);
 
drupal_set_message(t('Ctools people are crazy and wrong.'), 'error');
}
?>

Please, discussion of CTools

merlinofchaos's picture

Please, discussion of CTools merits flaws and design is off topic.

Agreed, just wondering why

pounard's picture

Agreed, just wondering why the most participating people are three CTools developers.

Pierre.

Because if the core plugin

merlinofchaos's picture

Because if the core plugin system ends up unsuitable, I have to maintain a parallel system into D8 and I'm against that. Every time core implements something that is only good for core I grind my teeth. I'm looking at you, dashboard.module.

Agreed, just wondering why

eaton's picture

Agreed, just wondering why the most participating people are three CTools developers.

Because collectively they have more experience building plugin systems, and thus more real-world knowledge of the ptifalls that have already been encountered and worked around, than any other developers in the Drupal ecosystem.

Earlier you mentioned the idea that "Version 1 is crap, version 2 is stable crap, version 3 is good." That only happens if everyone pays attention to versions 1 and 2; otherwise, you're blindly reinventing new systems and jumping back to the "1.0" stage. I'm by NO means married to CTools, and I'm well acquainted with its frustrations, but dismissing their input and brushing away code they're working on for possible use in core is foolish.

Just wrong

neclimdul's picture

You keep saying it was designed for procedural code but you are in fact wrong. It was not designed for code it was designed for data discovery. Your plugin can be code, strings, classes what ever. This seems to be and have been the core of your argument and I'm getting tired of saying it.

Fine, redesign. In my experimental sandbox I've re-factored almost every line and completely dropped features until they can be more clearly defined and re-implemented.

And being as you probably haven't been a part of the CTools plugin system development, its internals have been re-factored several times. Actually, the oldest and at one point almost deprecated part is the big define a bunch of plugins in a single hook part that core is going to require. Based on this you might say core would be holding CTool's design back. (core probably will make good use of the feature though so its worth keeping)

So, I know you said you didn't mean for it to be personal, but please stop calling a wealth of knowledge and experience crap and implying we're being inflexible about our implementation details. Its just not true.

I like spiking a bit, sorry

pounard's picture

I like spiking a bit, sorry for that :) I do know that you do good things (your actual D7 patches and CTools both carry good stuff).

Pierre.

Why B Wins

mfer's picture

We can talk about a lot of "what if" or "how we should do it" of this. But, option B is basically what you can do with ctools, right? This is tried and tested.

Option A is untested vaporware, right? Something that's an untested concept can't be corner tested. We don't know how it fairs to all the possible ways people will want to use it.

Since B is something we can do now, how would A be better than that? Better enough to throw out what's already proven?

Drupal 6, 7, etc does have

pwolanin's picture

Drupal 6, 7, etc does have plugins already. We call them modules and themes.

My initial take is that option B is better - seems like that's more likely to give us the ability to a lightweight request handling.

No

Crell's picture

Modules as they exist in all versions of Drupal up to and including Drupal 7 are not in any way shape or form related to plugins as discussed and described in this thread or in WSCCI in general. Drupal core currently has no equivalent to plugins whatsoever; contrib has a mix of ctools and lots of one-offs.

Request handling is something that would happen almost immediately after all three systems are initialized, as it requires configuration, context, and pluggability. Whatever the order we need all three first anyway, so that's a moot point.

Modules as they exist in all

merlinofchaos's picture

Modules as they exist in all versions of Drupal up to and including Drupal 7 are not in any way shape or form related to plugins as discussed and described in this thread or in WSCCI in general. Drupal core currently has no equivalent to plugins whatsoever; contrib has a mix of ctools and lots of one-offs.

This is my jaw hanging open in disbelief.

If that is your stance, then your idea of plugins is wrong, and I have no ability to reconcile that with reality.

Yes I don't really get that

catch's picture

Yes I don't really get that statement either.

We have various pluggable subsystems in core, some are better than others.

We do actually have one plugin system for a certain group of things - $conf['plugin_inc'] = DRUPAL_ROOT . '/path/to/plugin.inc'; - that works for session and lock at least.

Modules can be swapped out for a completely different implementation if you like - iirc it's entirely possible to create your own 'node' module, dump it in sites/all/modules and have it run instead of the one core ships with - that might be a rarely used feature but it exists.

They may architecturally be nothing to do with plugins as you're describing them, but they are things that are pluggable, and describe real use cases and implementations. If you ignore all this and other examples and say "not what I mean" then you are playing cheshire cat.

* The problem is how those

pounard's picture

* The problem is how those three pillars relate.
* We want the configuration system to be flexible, and allow for more than just one hard-coded implementation. That means it needs plugins.
* We need the plugin system to allow plugins to be configurable. That means it needs configuration.
* We need the plugin system to be able to respond based on more robust request information. That means it needs context.
* We need the context system to support swapping out what code is responsible for handling what piece of contextual information. That means it needs those handlers to be pluggable.
* We need to be able to configure what code is responsible for handling what piece of contextual information. That is, it needs configuration.

Hen or egg?

You will have, at sometime, to decide which comes first. I'd say that if interfaces comes first, configuration layer on top, you should be able to have a non circular dependency schema. You'll also need to make some concessions and hardcode stuff.

Could be:
1. Configuration system is basically interfaces of a mapping layer, init it at first place.
2. Bootstrap the database now, if needed. Database should have its own autoloader in the autoloader stack btw, to avoid the current messy D7 implementation (that lead me to some fatal errors because of some database loggers inited too soon).
3. Bootstrap the cache system (including cache backends) do not make cache backend use the plugin handling, it's an error. Cache backends should be hard-configured (cache backends remains pluggable, but do not need discovery since the sysadmin will configure them manually), never allow modules to change them it's being heretic! Cache backend belong to the sysadmin's architecture and is NOT business logic.
4. Database can be basically bootstrapped here instead of upper, but in order to use the database cache backend, you'll need it upper.
5. Some primary objects of the configuration/mapping layer are independent of the rest (variables implementation should NOT be pluggable, you will break Drupal to Drupal compatibility), and basic arbitrary array based configuration objects and volatile implementation can already be used from now!
6. You bootstrap the request object, request does not needs configuration, it provides some of it (can be bootstrapped at anytime, even way upper!). It can be pluggable, if you have a incoming request which is not HTTP, for example, it should be pragmatically derivated somewhere in the bootstrap, but it should'nt use the plugin system, request is mainly readonly stuff. You can eventually never bootstrap it manually (even better) and let the MVC (or PAC (or whatever)) do a lazzy instanciation depending upon what happened during bootstrap. If you choose it to be non plugin based, you can even boostrap it way upper and use it for the early cache page system.
7. Plugin uses the mapping layer interface to work, you can bootstrap the plugin system. From now, you can use basic plugins that uses the early and independent configuration objects.
8. Now that plugin system is bootstraped (and this is a REAL bootstrap by the way), you can finish the configuration bootstrap (if needed, but I really doubt there is anything that needs configuration objects to be plugins driven WTF would need that?). Plugins can use plugin based configuration layer specific objects!
8. Initialize the context system, primarely interface based, depending on configuration interfaces, as the plugin system has been bootstrapped, it's ok.
9. Then here you go, you can now allow modules to start using plugins with everything that exists, injecting contextes and stuff (here the "swapable context" comes into action).
10. Yay, run MVC (or PAC (or whatever)) (here the request can be lazzy bootstraped) Drupal is running!

Pierre.

I believe that my proposal

pounard's picture

I believe that my proposal totally implies option A. And BTW, option A is better. You need configuration from the very begining, whatever happens, so plugin system should depend upon configuration layer, and absolutely not the opposite (WTF would anyone do this seriously?).

Pierre.

You need configuration from

merlinofchaos's picture

You need configuration from the very begining, whatever happens, so plugin system should depend upon configuration layer, and absolutely not the opposite (WTF would anyone do this seriously?).

No. You don't. Why do people keep saying this?

So you actually do agree with

pounard's picture

So you actually do agree with me?

EDIT:

Sorry, I think this is because everyone read that (but not the rest of the paragraph):

That is, configuration is the bottom of the stack

Pierre.

If you actually want plugins

pounard's picture

If you actually want plugins to be configurable by context, and context being a specialized instances of configuration (mapping) objects, then you need the plugin DESIGN to rely onto the mapping layer. This is I think a non negociable statement if you want a common interface for plugin configuration.

Pierre.

IMO this is false. But since

merlinofchaos's picture

IMO this is false. But since it's non-negotiable, I'm just going to tell you you're wrong and leave it at that.

Except if you devide the

pounard's picture

Except if you devide the plugin layer in two layer, once none depending, and one which is. But my statement was a syllogism, it started with an hypothesys that might be untrue (the fact that we want configurable plgins, and that this configuration is in the design itself). BTW I supposed that Crell would want that, and I'm quite sure that I'd want that.

EDIT: The reason why I'd want that is that the plugin registries and or factories can the embrace the configuration since the design itself (the interfaces) and provide this as an helper that plugins might use or not, but it will make the whole API consistent.

It would make the design so much clearer, avoid a lot of code duplication and code indirection (a non configurable plugin layer, then a configurable plugin layer while all could be configurable by design seems a bit of code indirection to me).

Plus, it would virtually make all plugins, without exception, configurable with the lowest costs.

Pierre.

Can you explain to me how a

neclimdul's picture

Can you explain to me how a registry relies on configuration and context?

I'll go ahead and put this out there, if it doesn't plugins do not rely on configuration and context. An implementation of a plugin factory might, but plugins do not.

I believe in option

fabsor's picture

I believe in option A.

Creating one sane way of dealing with configuration sounds like the best starting point, even if it means we don't get to use a shiny plugin system for the basic tasks. For most people the configuration won't be a problem, and the people that are running complex environments has probably have solutions to the filesystem problems already anyway.

Having one way to do it (tm) is not as important as solving the real problems that we are dealing with in the early bootstrapping process and in the configuration management area in my opinion. Being the PHP programmer that I am, I choose solving problems that we are actually experiencing in userland rather than focusing on beautiful design. The computer science part of me screams in dismay over this conclusion however =)

//Fabian Sörqvist

This comment seems to

merlinofchaos's picture

This comment seems to directly contradict itself.

The problem I see with option A is precisely that it forces "one way to do it" on plugin design, and leaves out the ability to have plugins for the bootstrapping process itself.

Creating one sane way of dealing with configuration sounds like the best starting point

I don't like the way this is worded. It seems like that means option B is somehow insane.

Having lived in a world of plugins for several years I find this disheartening, to say the least.

There is no one way

mfer's picture

Creating one sane way of dealing with configuration sounds like the best starting point, even if it means we don't get to use a shiny plugin system for the basic tasks.

The web is a mix of different configuration systems. The more Drupal integrates with these other systems the better off we are in being able to make the configuration system pluggable. It would be nice for Greg Dunlap to weigh in on this as he has done a lot of research on these stuff.

Simple, Optimized, Realistic

ronald_istos's picture

Principles:
1. Simple to implement and understand - makes it easier to fix when broken
2. Optimized solutions for the problem at hand - makes for better performance and avoids the pitfalls of endless architecture discussion (and I do love a good architecture discussion)
3. Will give us Drupal 8 in the next 18 months

For the reasons above I like 2 and 3 from Crell's 4 options and the A approach overal.

Configuration and context handling are new and there is a lot to figure out beyond how the plugins will work. Let us not confuse matters further by trying to come up with the one true system and introducing dependencies purely for the sake of a single type of solution to each type of problem (one true config method using the one true plugin method using the one true context method).

Conceptually the system can be the same (i.e. follow similar principles and approaches) but implementation wise they can differ and diverge to suit needs and meet the end goals.

For me the end goals would be:

  1. A context system modules and core can use to simplify their tasks
  2. A config system that makes deployment a much better experience

After we have something working we can talk consolidation in terms of how the goals are achieved in Drupal 9 or whatever. With experience the choices by then might even be more obvious.

In the meantime module developers will benefit from a plugin system that is well suited to their needs and is not overly complicated because it has to do configuration and context stuff. Keep in mind that most developers will use what is available and not necessarily change the way it does things - just like most people really use MySQL and and not Mongo or anything else. So even if the plugin system is not the same as the one they can access via modules for other tasks that's fine for a bit - most will never know ;-)

Solid improvement in the overall systems can be huge for Drupal 8 and very beneficial to developers - let us make sure we don't miss that because in an attempt to not only overhaul everything but do it in a way that is overly ambitious. In that respect the very first discussions people in San Francisco were very well suited in that they were about phased implementation, ambitious in reach but realistic in steps to get there.

Configuration and context

merlinofchaos's picture

Configuration and context handling are new and there is a lot to figure out beyond how the plugins will work. Let us not confuse matters further by trying to come up with the one true system and introducing dependencies purely for the sake of a single type of solution to each type of problem (one true config method using the one true plugin method using the one true context method).

Another one that seems self-contradictory to me. I agree with this part:

Let us not confuse matters further by trying to come up with the one true system and introducing dependencies purely for the sake of a single type of solution

Then why do we want to force plugins to have a dependency? Option A forces plugins to be dependent on context AND configuration. :(

Alternative plugin systems

ronald_istos's picture

We are forcing the plugin system that module developers will use to depend on the configuration and context systems - we are not forcing the plugin system that configuration and context will use to depend on the same solution that will be appropriate for module developers.

That is the difference:

  1. Focus on configuration - don't (relatively) worry about the plugin system it will use
  2. Focus on context - don't (relatively) worry about the plugin system it will use
  3. Focus on the plugin system (that 95% of devs will use as opposed to the plugin systems above) whilst offers consistent access to configuration and context that 95% of devs will use.

Does that make more sense or am I just missing something?

That doesn't pick option A or

merlinofchaos's picture

That doesn't pick option A or B.

The fundamental choice between A and B, as I stated http://groups.drupal.org/node/150149#comment-498534 is that in option A, plugins have an unbreakable dependency on context and configuration, and option B, they do not.

Personally, I do not understand why we would force that dependency upon them. By forcing that dependency on them, we will have to create a different system, that does fundamentally the same stuff, to be able to swap out anything that comes earlier than the plugin system. Why? Because we're forcing a dependency, even though most people seem to be saying we do not want to force a dependency.

So even if we don't worry about what plugin system config or context will use, I still fundamentally disagree that we should force that dependency on plugins. And nobody who wants to force that dependency has been able to state why, other than pounard's (IMO misguided) belief that configuration is somehow fundamentally necessary for them, when it's not.

Please note that by not forcing that dpendency, I do not mean that we leave it a wild west where all plugins have to figure out their own path. Most plugins will be able to use the same code. But we just make it optional. It becomes the difference between the base plugin class that has context and configuration and the one that doesn't.

Let me restate that going

EclipseGc's picture

Let me restate that going with option A as opposed to B only cripples the system plugins are supposedly depending on here. There is no good reason what so ever to depend on config or context. The plugin system can be written in such a way as to provide config/context-less plugins, and extended as necessary for each of these other major pillars availability.

I.E.

  1. We can have a base plugin system class that provides the ability to create config/context-less plugins.
  2. We extend that system once configuration management is available and provide (for context) plugins that have access to configuration (context arguably can't do anything before that anyway so the point is moot what access it's plugins have... they're the same level it is
  3. Extend the class again once context is available to provide configuration and context information for all subsequent plugin types

Bottom line, we don't have to pick... I don't know why the argument is even framed this way.

Eclipse

Implement context & configuration as plugins

bsenftner's picture

This strikes me as the correct way to frame this situation and solution.

Simple: context-less, configurable-less plugins at the logical bottom, and at some layer above a plugin for context and a plugin or configuration or a single plugin that implements both. Either way, we get want we need, without some complex environment at stage 0. Keeping complexity as low as possible is a very good thing.

(Drupal module dev: 2 years; developer for 33 years including OS dev for PSX, & other embeddable OS environments architect of multiple media production environments & 3D simulation engines.)

Love your signature, I

pounard's picture

Love your signature, I personally never judge people experience before I actually talked to them or read what they've done.

EDIT: Your comment might be good, just for this reason I -1 it.

Pierre.

By the way: everyone agreeing

merlinofchaos's picture

By the way: everyone agreeing with Larry on option A, please stop and ask yourself what that really means.

Personally, I think Larry is vastly overstating the "complexity" of what it means to have a plugin system that can support context/configuration optionally. That's really the only difference.

In proposal B, plugins can live without context or configuration.

In proposal A, plugins must have context and configuration.

Why? I don't know. I don't understand why this is a requirement and a dependency. In my world of plugins, it totally is not a dependency. I can have plugins that simply don't need to be configured. Alternatively, I can have plugins whose configuration is so simple that we can skip the more complex configuration interface and just read a value from settings.php so that we can bootstrap the rest of the system on top of it.

And that is, as far as I know, the only difference. I think that the desire to force this dependency on plugins is irrational, and that somehow Larry has convinced himself (and apparently a bunch of people) that this requirement is so serious that building a plugin that doesn't utilize context or configuration will somehow so confuse the situation that the system will be hard to use.

Please remember that I've been working with plugins in Drupal since 2007, and I've seen a lot of use cases and a lot of ways they can be used, and done a lot of experimenting, and I believe that it is totally possible to have a good plugin system with optional pieces that is easy to use and does not require a dependency on context or configuration. Having those two pieces as something that can be optionally added in, by using a richer plugin class or interface is, IMO, completely the way to go.

And I really don't want to come back to this 2 years ago and say "I told you so" when everybody working on this catches up to the things that we've already experienced and goes "Oh. Yeah. You were right."

To me Proposal A does not

ronald_istos's picture

To me Proposal A does not mean that plugins must use configuration and context it means that if they have to use it they will in a consistent manner. If they must have it then I am with you - I don't get why.

Having said that - you may be right that the complexity of a plugin system may be overrated. I am coming from the point of view of someone who has not had to do it in Drupal but knows that in general getting such things right is not straightforward - you are coming from the point of you of someone who has done it in Drupal (a few times over) so trust your opinion more than mine on this.

Maybe we can have our cake and eat it?

Yes, exactly, the A proposal

EclipseGc's picture

Yes, exactly, the A proposal is that plugins won't even exist until config and context and bootstrapped. Proposal B says config and context and benefit from plugins, so start there... the "consistency" of a plugin's available configuration/context will be related (essentially) to the level of bootstrapping that's been done before that plugin needs to fire. If it's a plugin for configuration management, it won't have that stuff, if it's for context, it'll only have configuration, and if it's after both of those things are available, then it can have both.

IMO, yes, I think we can have

merlinofchaos's picture

IMO, yes, I think we can have our cake and eat it. Because IMO, the big difference is that we have a Plugin Interface with the minimal funcitonality necessary to operate, and another one (or more) which introduces context/config as necessary.

Then, when creating a type of plugin, the proper interface is chosen, and those plugins will all behave consistently. I do not think this is particularly complex, and that's part of my objection to the initial argument. This is, IMO, proper abstraction.

This is, indeed IYO.

pounard's picture

This is, indeed IYO.

EDIT: I do agree with you this is doable. If you need decoupling between the configuration layer and the plugin layer, this is the good way to go. But refering to #498759 and #498779 (which are my own posts on this thread) and to the fact that the goal of those two core initiative is in fact to couple those systems to ensure maximum efficienty, simplest design possible, and performances while ensuring a fully featured toolbox, I really think this isn't the good way to go.

The ideal of the whole, and the reason why this original Crell's post is also on the configuration initiative group is because both of those systems needs to be coupled to avoid code indirection and code duplication, and to provide a really, really powerful CORE toolbox (by toolbox I mean that users totally have the RIGHT not to use everything in it).

Pierre.

"simplest design possible" !=

sdboyer's picture

"simplest design possible" != simplest design actually needed for all uses cases != simplest design we can come up with in forumspew.

Why oh why are people so hell-bent on cutting ourselves off at the knees? We (ctools folks) have done plugins in Drupal for years, and have a feel for what sort of variety of needs, styles, and approaches there are out there. Most of the rest of the folks in here have just tinkered with it, or have built their own one-off use cases. Why oh why is it so hard to believe us when we say, "look, this is complicated. Please don't lock it down too far prematurely."

It's not about whether you like how ctools works. It's about whether or not you believe the variety of plugin use cases we've seen should have bearing on a conversation about the future of plugins in Drupal.

If you have so many use

pounard's picture

If you have so many use cases, why do they all being called plugin, no variation?

Pierre.

So is what you're saying that

joachim's picture

So is what you're saying that with option B, plugins can go both before and after C&C?

In other words, the bootstrapping process goes:

  1. config-less plugins
  2. context & config
  3. plugins that need context

and your hierarchy of basic plugin classes would go:

  • PluginNoConfig
    -- (your classes for step 1 here)
    -- PluginWithConfig
    --- (all other plugins here)

That seems to me like a pretty neat way of having your cake and eating it :)

I think that if we want to

oskar_calvo's picture

I think that if we want to build a good drupal 9 it should be re-do.

There are parts of drupal 5 in drupal 6 and drupal 7, and there are no sense to have them.

Also, I think the best idea is a really tiny small-core, and them everyone should install or develop what they need.

oskar

There are parts of drupal 5

pounard's picture

There are parts of drupal 5 in drupal 6 and drupal 7, and there are no sense to have them.

Totally agree with you on this.

I think that if we want to build a good drupal 9 it should be re-do.

The idea of this is in fact to remove any D5, 6 and 7 code from plugins and configuration layer, and this wouldn't need to be redone for D9 if it's done well.

Also, I think the best idea is a really tiny small-core, and them everyone should install or develop what they need.

Drupal is a CMS that is totally forgeting about it's primary goal: manage content. This is a shame. If Drupal starts providing a tiny core, you can be sure that I won't be part of it anymore, if so, I would use Plone as CMS, and Zend Framework or Django as framework, since they are really more advanced on many many things than Drupal. What Drupal have and does is content (nodes) and it should probably kept that way (even if nodes are basically derivating into entities, entities for the most still are content).

Pierre.

pounard, I think that Drupal

oskar_calvo's picture

pounard, I think that Drupal should be a really small core, and you can choose what you need to install.

A small core should able to manage entity (create,edit, delete), blocks, and one login system, but I think the login system should be plung and play, When I have to build another ways to make other kind of login I suffer.

There are a lot of modules in the core that should be out, and people decide what they want to install.

Oskar

I don't believe in "3

donquixote's picture

I don't believe in "3 pillars".

Consider a plugin object. On instantiation, this object receives a bunch of behavioral options. From the perspective of this plugin object, these constructor params are the "configuration".

Where was the plugin created?
The plugin itself does not care. It has its configuration, telling it how to behave.
In fact the plugin was created by a plugin factory function (or method).

The factory has assembled the plugin configuration from various other data: Maybe it received a string key as an argument, and then looked into the database (or something else) to retrieve the configuration for the to-be-created plugin object. The db row containing the plugin configuration might be just one of hundreds of similar rows, representing different possible plugin object configurations.

The factory is part of a module, which defines its own way of storing data. There are good reasons why the module developer decided to use a custom db table, not some standardized configuration storage.

Maybe the same module has another type of plugin, which is needed at bootstrap time. The factory that creates those will not look in the database, but it will look at
- one site configuration variable defined by core
- another configuration variable defined by the module itself (both using the efficient filesystem storage that we are so keen about)
- in addition, it will look at some values in the request (such as, what subdomain we are in)
- and maybe some param in the hook that has triggered the factory. We don't know where this comes from.

From these values, it will compute something and decide which plugin class to instantiate, what should be the constructor params, and how many of these plugins it wants to create.

Instead of the 3 pillars, I'd say:
From a "global" perspective, we have
- different persistence systems. One is the new filesystem-based configuration storage, which is available early at bootstrap. Some part of this can be loaded all at once, other parts can be loaded when requested. The other is the database (or even, one or more databases). Modules somehow are given gateways to each of these persistence systems. Each of these 2 systems can be hacked by a developer/admin (that is, editing values without using a web-based configuration form), but we want that hacking the filesystem-based system is easier.
- data that depends on the current request. i'm not sure if this is 100% the same as "context". There is some raw data that really only depends on the current request. And then there are some processed data such as current user, current language, which depend on request data and configuration / persistence data. We want that anything depending on db persistence comes later in the request.

(btw, i think that "request" should not have to be identical with "php process")

From the perspective of a single plugin, we just have constructor params (totally ignorant where these come from), and/or maybe some setter params (although i prefer constructor). These can be a mix of request data, persistence data and other plugin objects. Maybe even some random (like a randomized color plugin).

Plugins (or just any objects) can be created at different times in the request life cycle, and the plugin factories can be fired in various situations, with different parameters involved.

Trying to seperate the system into plugins and configuration, is like putting all food in one bucket, and all packaging in another bucket.

The configuration "pillar" is really just a nice way of storage, for situations where the database is not a good fit. We provide a nice gateway object, and make sure that it is made available where it is needed. Ideally by passing it around.

If we want a "unified plugin system", fine, but rather like a toolkit than a framework.

No "Plugin Pillar", no "Plugin Layer"

donquixote's picture

And that said, I also don't believe in "plugin layer" or "configuration layer".
Plugins exist on different layers, and configuration exists on different layers.

As Merlinofchaos already said, modules (with their hooks) are equally valid examples for plugins, as are db engines, cache engines, input formats, etc. The implementation of these is and has to be totally different.

For instance,
- some boot modules and their enabled/disabled status could depend on some early-bootstrap config file
- another mid-bootstrap system of config files could depend on a handler provided by some of these boot modules
- the rest of the modules' enabled/disabled status would depend on a config file AND a throttle setting (also in a config file)
- the db engine would be provided by one of the modules
- another piece of configuration would be dynamically loaded from the database
- some page-level plugins provided by modules would depend on the request (=context?) and the database, and the theme chosen for this page.
- an input filter plugin would depend on the piece of content we want to render.

I don't see any place for a generic "plugin layer" in this scenario.
There are multiple layers of configuration - if you want, we could name one of them THE configuration layer. But what is the benefit?

4. Allow context to be

donquixote's picture

4. Allow context to be optional for plugins, based on their type.
* Pros: Plugin types that don't need context don't have to deal with them. We can use plugins for context handlers because we then don't have a circular dependency, so there's more consistency.
* Cons: Adds complexity to the plugins system, since there are multiple code paths. You may define a plugin type to not need context, but then oops, one particular implementation may need it and then you need to try and hack your way out of that hole.

5. Allow configuration to be optional for plugins, based on their type.
* Pros: Plugin types that don't need configuration don't have to deal with them. We can use plugins for configuration, because then we don't have a circular dependency, which allows configuration itself to be swappable.
* Cons: Adds complexity to the plugins system, since there are multiple code paths. You may define a plugin type to not need configuration but then oops, one particular implementation may need it. (We saw this with field formatters in Drupal 6.) Still doesn't solve the problem of how you configure which plugin to use for configuration.

The "Cons" all come from the idea that "a plugin is a plugin".
Life gets much easier, if we simply accept that an early bootstrap level plugin needs to have zero in common with a late-game tier-3 plugin.
And even in the late game, modules should not be required to use all the same architecture for their plugin systems.

We can provide one "make many people happy" plugin architecture in core, that will then be used in 80% of all plugin-related situations - fine. But this does not mean we have to use this same architecture for early bootstrap "pluggable configuration storage".

Factories...

donquixote's picture

Btw, why am I the only one to use the term "factory" ? How do you guys want to create your plugins objects then?

No absolutely not, I tried to

pounard's picture

No absolutely not, I tried to say it out loud there: http://drupal.org/project/issues/1112184 (and other many, many, many other places) since the begining of this group

Pierre.

True, some mention of

donquixote's picture

True, some mention of "factory" in there :)
I did not notice these posts. I did not even know about the "Plugins in core" thing.

And I think there is something else, that is "negotiation". If different modules provide plugins or plugin factories, then how do we choose? Do we let the factories create their plugins first, and then discard some of them? Or do we discard some factories? Do we let other modules mess around with the result aka *_plugins_alter() ? What information do we provide to these modules involved in the negotiation process?

In some cases it might be sufficient to have one configuration variable, or a value associated with a piece of content, that decides which plugin to use, or which factory to ask to get the plugin. In other cases, we have to ask a bunch of modules, based on dynamic parameters.

I think this negotiation and factory part, and how modules can register their plugins, is really the challenge. And this is so situation-specific that I can hardly imagine a one-size-fits-all system. The rest is just, inject object A in object B, done. This part does not need a unified plugin system, that's just the usual thing to do in decent OOP.

Another thing I'd like to do

merlinofchaos's picture

Another thing I'd like to do is take a solid look on what this theoretical dependency means. I'm going to lightly discuss 3 plugin concepts that are either in use today or we would very much consider using a plugin system for. We have some issues talking here, because the configuration system itself is pretty undefined, so it's hard to pin down what a dependency on this really means. But I'm going to try anyway.

Configuration means, to a plugin, acquiring the data that plugin needs to function, and getting that data to the code (presumably an object) that represents the functionality of the plugin. There are two important pieces to configuration:

The data itself, and the storage of said data.

When speaking of non-negotiable rules, for me this one is non-negotiable: the itself plugin should not be involved in the storage of the data. In fact, I don't think the plugin system should be involved on it. Why?

Scenario 1: The database plugin

It turns out, how we handle databases is more or less a plugin, though it's not currently formalized that way. But we have the process of selecting which one to use and the configuration for it.

However, it happens so early in the system that we have only one choice for configuring it: our settings.php file. Also, we can have more than one database plugin active, so it isn't simply select one and that's the only one you use. Configuration for alternative database plugins might be stored somewhere else.

A plugin system would be responsible for collating what database systems we have available, and using the values in settings.php, selecting the right plugin, giving it the data, and handing control of database functions off to the plugin. Most of our db_* functions are really just wrappers around a bit of code that selects the right open connection and passing off to it. It's very nearly a canonical example of what a plugin is supposed to do.

Is this system dependent on configuration? Yes, in the sense that we need enough configuration to select which one and how to configure it. But is the plugin dependent upon this? From the perspective of the plugin, all it knows and all it should know is that it was retrieved and given some configuration data. In the base case it was in settings.php, but for some module or something adding an alternative database, maybe it put the configuration somewhere else. But the plugin does not care. It is not dependent upon the storage of the configuration only the data. The general flow looks something like this (total pseudocode, this isn't anywhere close to real):

<?php
 
// Scenario 1, plugin factory, config external
 
global $conf;
 
$db = plugin_get_object('system', 'database', $conf['db_type']);
 
$db->init($conf['db_config']);
?>

There's about a hundred ways I could write that that are all the same:

<?php
 
// Scenario 2, static OO factory, config external
 
global $conf;
 
$db = plugin::getObject('system', 'database', $conf['db_type'], $conf['db_config']);
?>

Or

<?php
 
// Scenario 3, plugin controller factory, then plugin factory, config external
 
global $conf;
 
$controller = plugin_get_controller('system', 'database');
 
$controller->getObject($conf['db_type'], $conf['db_config']);
?>

Iterating more on that is left as an exercise.

So is the plugin dependent upon configuration? It's certainly dependent upon the data but it should not be dependent upon where that data lives or how it's configured. Dependent on configuration means this:

<?php
 
// Scenario 4: configuration dependency.
 
$db = plugin::getObject('system', 'database');
?>

And the plugin system magically figures out which database plugin to retrieve and configures it. Now we're dependent upon configuration, but we've also limited ourselves. Why does the plugin system know where this data lives? That is a tight coupling that violates principles of abstraction. Now the plugin system has to know where that data came from.

Scenario 2: Imagecache presets

Imagecache presets are a great example of a list of plugins, each plugin is uniquely configured, and they are all run in order. At the end, there is a result. To pseudocode what this looks like, we have:

<?php
// Scenario 1: pretend to process an imagecache preset, ignore plugins
function imagecache_run($preset, $image);
  foreach (
$preset->actions as $action) {
   
$action->run($image);
  }

  return
$image;
}
?>

Now, in this scenario, a $preset is an independent object. It contains the data for the list of presets, the order they are run, and how each preset is configured. See where I'm going here? Maybe we have something like this:

<?php
// Scenario 2: Process an imagecache preset using plugins
foreach ($preset->action as $action) {
 
$plugin = plugin::getObject('imagecache', 'action', $action['plugin'], $action['conf']);
 
$plugin->run($image);
}
?>

Again, the plugin is dependent upon the configuration data, but it is not dependent upon the storage mechanism of the configuration. There is no global place to retrieve the configuration from. The more the plugin has to know about where the configuration comes from, the more tightly coupled it is to a system, the harder it will be to test in isolation, and the more difficult it will be to get data to the plugin.

Scenario 3: A layout plugin

In Panels, I have layout plugins. A layout plugin looks a lot like page.tpl.php, in that it contains a definition of the regions that the layout contains, and a template that renders the layout with the regions. Let's provide a simplified example:

<?php
/**
* @param $layout
*    The identifier of the layout plugin.
* @param $content
*    An array of content which is keyed by region. Much like in page.tpl.php in Drupal 7.
*/
function panels_render_layout($layout, $content) {
 
$plugin = plugin::getObject('panels', 'layout', $layout);
  return
$plugin->render($content);
}
?>

As you can see, there's no configuration necessary. Now, in a brave new world of context, it sure would be great to inject context into this somewhere, so that the template could do stuff based on data. Right now it has to be ignorant of data.

But it does need data. It needs to know what to render. This is a little like configuration, but not really. In theory I can render several different layouts with the same instantiated plugin, but with configuration the theory is that if I want a plugin configured differently I need to instantiate a new one.

Bonus scenario: Context plugins

Wouldn't it be nice if we could just do this:

<?php
// Scenario 1: Retrieve an HTTP context plugin for a basic page request.
$context = plugin::getObject('system', 'context', 'http');
$context->init($_SERVER);
?>

Or

<?php
// Scenario 2: Retrieve an drush context plugin for a drush command
$context = plugin::getObject('system', 'context', 'drush');
$context->init($argv);
?>

Approximative quote: There

pounard's picture

Approximative quote:

There is two kind of objects, the ones that do something, and the ones that create the one that do something.

Your code is invalid, you are not using objects but static methods. Static methods should probably definitely kept for singleton pattern implementation.

Just messing up a bit. But second category of objects are FACTORIES and this is the right moment that everyone should start using generally known pattern names instead of reinventing the existing ones without knowing it.

May I quote myself about this topic:

drupal_static() is a procedural and basic (but efficient) implementation of the registry pattern, it should be named drupal_registry().

Start naming things. Plugins are more like "components" since you don't deactivate it, while module are plugins, but module is also the right word. The method you describe here are essentially static helpers for accessing specific object registries (which implies a discovery mecanism), and create those objects (which implies that those helpers are factories), Discovery imply heavy runtime operations, most of it should be cached, while you are refering to early bootstrap, where cache probably isn't bootstrapped. This is deffective by design.

Still messing up a bit.

Pierre.

Your code is invalid, you are

merlinofchaos's picture

Your code is invalid, you are not using objects but static methods

Yeah, uh. You're making huge assumptions about what's behind the pseudcode.

<?php
 
// Scenario 3, plugin controller factory, then plugin factory, config external
 
global $conf;
 
$controller = plugin_get_controller('system', 'database');
 
$controller->getObject($conf['db_type'], $conf['db_config']);
?>

This variant has no static methods, for sure. And, frankly, the methodology used behind the factory is an implementation detail and unimportant here. That's why I showed 3 different possible ways to do it.

If you are serious about not

donquixote's picture

If you are serious about not using static, it would be

<?php
function make_me_a_database($conf, $controller) {
 
$controller->getObject($conf['db_type'], $conf['db_config']);
}
?>

Next question would be, who calls make_me_a_database(), and how does this piece of code get the $conf and $controller?
Which inevitably leads to DPI or ServiceLocator (not as an argument, just as a starting point for further reading).

Let's not bikeshed the

merlinofchaos's picture

Let's not bikeshed the factory, ok? :)

Start naming things. Plugins

merlinofchaos's picture

Start naming things. Plugins are more like "components" since you don't deactivate it

Eh, I don't know if I agree with this.

I like what Peter said above. I think modules and themes are plugins. They are configured to be active or not; even in the inactive state, they are present on your system, but they are not given the opportunity to respond to events (or have their .module file loaded) if they are not active.

But to me, a module represents the essence of what plugins are: A bundle of functionality grouped together for common cause that can be added and removed from the system. We need code to find them, we need code to list them. We don't actually need objects at all for modules. (Yes there's a big argument about the benefits, but Drupal has gone 10 years without objects for modules. Definitely do not NEED them.) From a system perspective, about the only thing that the system that finds and loads modules depends on is the filesystem and the database where it caches available modules, filenames and paths and some state information. I think it's a good model, even if the implementation used is mostly wrong.

Two basic things I believe

sdboyer's picture

Two basic things I believe about this:

  • The distinction between A and B is overblown. There can be a middle ground.
  • The only truly dangerous thing here is an overly narrow definition of plugins. Only if we adhere to such a narrow definition is the situation dire.

In other words, I believe the direness of this situation is self-perpetuated by Larry's restrictive picture of what ought to constitute a plugin. We all agree that we want a system that is as simple as possible, but no simpler. If you really adhere to that principle, then when someone presents you with a situation your simple solution does not cover (as Earl has often done, and done again in [#498524] with the template plugin), the more correct answer is, "Ah, then my solution must be simpler than is necessary," not "Your example is invalid."

DX is obviously a very important point here - but we don't achieve that by reducing complexity or choice as a kneejerk rule. UX comes from a logical organization of information & choice. We can have a multi-layered plugin system without it being a snaky mess of ifs and indecipherable indirection.

We want to learn from other systems? Let's do, please: symfony2's "plugin" system, their bundles, is little more restrictive than something that autoloads, plus a dependency injection container. And they do benefit - a lot - from the fact that their bundles aren't a class that must implement an interface, but rather a collection of logic that is made available to the system in a standard way. Hell, they can even plug code in directly from other frameworks.

Finally, we can solve this problem with a layer of indirection, even assuming we go with the tight restrictions on the nature of plugins that Larry is advocating. Make plugins the base system on which all else depends, but make the base plugin interface/pattern expect a service container (which can be hardcoded and be the first thing loaded after settings.php) from which it can retrieve its configuration, instead expecting literal config. That service container is then populated with additional capabilities as bootstrap continues and more services become available, so they implicitly become available to the already-built plugins. But that service container can be injected into pluggable conf readers/context just as easily as it can be injected into a session or cache plugin.

Other frameworks and DI

mfer's picture

I like what you are pointing out here.

First, it would be nice to use the plugins / class of other systems. There have been cases where parts of Zend have been used within Drupal. It would be nice if we could natively use them within our plugin system rather than have to make wrappers for them.

Second, I like the idea of having a container that manages plugins and dependencies but gains more to it through the bootstrap process. Something along the lines of http://pimple-project.org (yes this is a real project). I could see this container providing a nice manager for configuration, context, plugins while making as much pluggable as possible.

100% support #A

Sylvain Lecoy's picture

I think the package #B is redoing the same mistakes we've done with Drupal. Having an optional configuration is, to me, the same problem as loading everything which is enabled and let's do shit with it. Having an optional context will keep preventing Drupal to figure out whether to load a cached page or not for instance.

Having the context and configuration layer as start level could solve the issue of "let's load all shit" for a simple request that's using less than the quarter of the code loaded. Maybe some plugins wont need a configuration or a context, but I bet in the global, there will be at least one plugin needing a context to answer back the user. So the earlier its loaded the better it is.

Let's think a bit about WebServices as well. Without configuration, obviously without context as first ingredient, you can't know for sure how to answer back: is it JSON ? is it AMF3.0 ? This is the required platform to bootstrap correctly. Not having the context as input is like doing thing without really knowing why, it seems to me a blindly but easy approach, we should'nt be doing that. I know people will throw the golden hammer advocating using their growing product because they are proven to work well but we should expand our horizons and not being narrow-minded. For sure using something that already exists will minimize risk but requirements are not fully met in CTools, and it shouldn't dictate design and system architecture.

1 and 2 seems the right way to do it, 100% support #A.

EDIT: I see discussions here confusing me about the word "configuration". Sometimes its taken as configuration in component theory (which is a configuration of components, connectors, and configurations), sometimes as a way to store module parameters, and configurable variables. Maybe this is a big source of misunderstanding, as for the mis-used "factory" word as pounard highlighted.

EDIT: I'm speaking about configuration of plugins (how they work together), not their actual configurations (let's call it settings ?), which can be sorted however it need to be.

Here is a concrete example

Sylvain Lecoy's picture

Here is a concrete example with pseudo code. I have this page, which in a block displays tweets about my website. To do this, i'm using a custom module depending on an OAuth implementation. Since my webserver have the PECL native OAuth extension installed, the plugin used gonna wrap this PECL implementation, but that configuration will be done by my sysadmin and as a developer I should not think about il.

Great, because the configuration knows (thanks to the context) that this page would requires this OAuth implementation.

<?php /* @file: twitter.module */

global $twitter_oauth; // the PECL plugin get injected with its specific settings (e.g. consumer_key and consumer_secret).

function twitter_favorites() {
 
$twitter_oauth->setCaller($uid_admin);
 
$twitter_oauth->fetch("https://api.twitter.com/~/favorites");
  return
$twitter_oauth->lastResponse;
}

?>

Without configuration this is not possible, i'm not speaking about settings, but the piece of code that makes my dependency to OAuth resolved. If moreover the OAuth object can be injected pre configured (with my plugin settings for this mapping) then it would be awesome.

For testing this makes even more sense, we just inject an async-stub plugin by swapping the configuration. The low-API layer knows that we are in a testing context and instantiate the right plugin. Again, this configuration is done by the sysadmin but obviously the developer can be the sysadmin.

Yes but the piece of code

merlinofchaos's picture

Yes but the piece of code that makes your dependency to oauth has to know a lot of information:

It has to know where it's stored and how it's stored.

And while your example works for the scenario 1 I posted earlier, it makes scenarios 2 and 3 totally impossible. I call that short-sighted at best and naive at worst.

BTW: I had this discussion with Larry 3 years ago, and it's kind of frustrating. Magically finding data linkage seems nice, but closely tying system to storage violates abstraction and makes testing hard.

Now, there's no reason that oath can't make a factory on top of the plugin system that adds the 2 extra lines of code needed to go fetch the configuration and then pass it to the plugin. Then, hey, systems that don't need or can't use that automated linkage won't be burded with it, or restricted by this unnecessary dependency.

It has to know where it's

Sylvain Lecoy's picture

It has to know where it's stored and how it's stored.

merlin, what is wrong with that ? It's the configuration API's job. If you make a factory on top, it stills need to know where it's stored and how isn't it ? For testing, well, It depends on the testing method, you can inject a Stub object by the configuration. No need to swap storage if your stub object uses mocked methods.

OK, there's NOTHING wrong

EclipseGc's picture

OK, there's NOTHING wrong with it. What's wrong is that option A prohibits the existence of plugins that don't have access to this information, and we're saying "DANGER DANGER!". Between merlin and the rest of the ctools people we probably have a decade plus of experience (collectively) dealing with plugins within drupal. What we're saying, as loudly as we can possibly do it, is that plugins don't NEED configuration and context. More importantly, this imaginary plugin that was defined that doesn't have context and suddenly needs it, IS COMPLETELY MYTHICAL AND BUNK. It's a trojan horse of an argument. They don't exist!!!

Plugins that DON'T have configuration will be plugins that are necessary before configuration is loaded. Obviously they won't ever have or need the configuration system. They'll be designed that way. Plugins that don't have context will only exist before the context system is loaded... again obviously they'll never need context because they'll be designed this way. If they ever need these things (for whatever reason) it's a limitation of the system, and one that I MUST POINT OUT is not imposed by the plugin system. All plugins after that can have access to both systems.

The question that Larry is posing isn't "Who thinks all plugins should get context and configuration loaded into them?" the question Larry's asking is "Who thinks we should prevent plugins that are smart enough to work without context/config from existing?"

So let's try this again... A or B?

My proposal on this comment

pounard's picture

My proposal on this comment totally fix the problem. It made plugin being aware of configuration objects API by design without the need of variables, database, context or whatsoever.

And from this comment prospective, plugins are configurable, but can also work without it as neutral objects, and does not rely on anything than a basic interface (the configuration object).

Pierre.

I think you are falling in

Sylvain Lecoy's picture

I think you are falling in the trap of misunderstanding configuration and plugin configuration (that I call settings).

I'm not speaking about a plugin configuration, but a configuration of plugins. Speaking about decades of experience is not an argument if you don't even understand the difference underlined here. See: Configuration VS. Settings in my post bellow the Pierre's post.

I'm trying to put this gently...

EclipseGc's picture

The problem is that you don't understand what we're discussing. The question isn't:

"Should plugins have configuration?"

The question is:

"Should the entire global configuration system be loaded and available to all plugins?"

For which I resoundingly answer NO! There are plenty of good examples of plugins that don't need any sort of configuration... cache and session just to name two (and frankly a huge portion of current menu architecture could be replaced by plugins that generally don't need configuration either).

Also, as a logic puzzle, how do I get local context if I don't already have global? Sure we can mock it, but that's essentially "config-less".

You are over-estimating

Sylvain Lecoy's picture

You are over-estimating configuration system, why adding complexity and size by your wording ?

I well understood the question, and my answer is yes, we need it if we want to do thing properly. That is, not loading every shit that is registered, but only loading a configuration of plugins.

Cache and session doesn't need plugin configuration, this is true, but they need to be loaded from a configuration. Otherwise, how would you know that you need an APC cache plugin or a MEMCACHE plugin ?

Totally agree with the fact

pounard's picture

Totally agree with the fact that the plugin system doesn't need the entire global configuration, I think that's not the point of this thread.

Pierre.

Right, that's what I keep

EclipseGc's picture

Right, that's what I keep saying. It IS in fact a big point of this thread. If we can all agree that plugins should NOT be dependent on the global configuration, then I think we've largely ruled out the "Option A" scenario. I think that point wasn't immediately obvious to most participants here, and that's why I've been spending so much energy on in this thread. The fundamental understanding to even have this argument hasn't been granted to all participants, and until it is, no one can have a meaningful discussion about this topic.

I think that if you replace

pounard's picture

I think that if you replace configuration per independent base mapping layer, without variables in scenario A, the description remains exactly the same, except it doesn't fit to #1 and #2 anymore, because you have both plugins with configuration in design and no dependency to any other pluggable system, then you can use configuration plugins that accept low level configuration at boostsrap, then plug the real variable system backend and then use the exact same plugin API but passing through complex plugable configuration objects to them instead of basic implementations from the mapping layer. That's all the stuff I described in my diagram.

In fact it's pretty much:
1. One and only one Plugin consistent API, still highly extensible
2. Configuration in design for plugins
3. No circular dependencies
4. Specific implementations of configuration can be pluggable.

And that's what I'd call "Scenario Z", pretty much all the Pros of the A, B and C scenarios:
1. Plugins are dependent on one and only one interface (by design) which is itself only dependent on SPL, per default on all PHP environments now. Which makes them independent for almost all the world (even from Drupal itself!).
2. System is pretty lightweight, but remains highly extensible.
3. Plugins that don't need configuration can just ignore it. No pitfall here, no performance impact.

While removing almost all the Cons, all in fact that are being described in #1, #2, #3, #4 and #5.

Further more, with a generic registry/factory, you can easily implement:
1. A common registry interface.
2. Generic and independent discovery mecanism (file based, hook based, globals based, ...) for bootstrap time
3. Specialized discovery mecanisms implementations you can use as soon as the rest is bootstrapped, under the exact same interface! One API to rule them all (I like neclimdul's code).
4. A default factory, that really doesn't care about what's inside the registry as soon as it implement the 1. registry interface.

It induces a smaller bootstrap adaptation, but that's OK I guess since it remains low level core stuff:
1. Bootstrap mapping layer base.
2. Bootstrap plugin system, use basics implementations of configuration object interfaces for configuration (Memcache cache backend for example needs configuration). Here you can already use some of the discovery mecanism.
3. Bootstrap high-level pluggable configuration objects implementations (such as variables handling, and probably contextes themselves could be a specialization of a low level mapping layer).
3b. Run some kind of signal that allows any module to init, bootstrap, use their high level plugins (we don't even need this, most of these module won't be bootstrap modules anyway).
5. Your Drupal is pretty much up for anything (still remaining localization and other core stuff, but that's not the problem here, and I don't really think it would impact them actually)!

Pierre.

I think we just disagree on

Sylvain Lecoy's picture

I think we just disagree on the fact that for you a dependency mean dependency injection by construction, while for me its the infrastructure needed to load the plugins (named mapping layer).

I agree with you because you are speaking on the plugin configuration, which is for me its settings, context should not be a dependency (in the constructor) as well since it will makes plugins dependant. Also, you can name the participants in question, always better than speaking generally about some black beast to hunt down until they accept the view point which seems right for you.

I will refere to my post abut this confusion: http://groups.drupal.org/node/150149#comment-499514 where I quote myself:

That's why, in my opinion, the configuration need to be the bottom of the stack. Without configuration (mapping layer, or whatever), you don't know what plugin to load. Without context, you don't know which configuration to load. And this is very important to understand that this doesn't mean configuration and context should be injected by construction in the plugins, which is a terrible mistake, as CTools folks underlined.

I am also quoting from Catch who wrote up a very nice sum-up about the way Drupal bootstrap (http://groups.drupal.org/node/150149#comment-499354)

index.php defines DRUPAL_ROOT to get_cwd() then does a require_once of includes/bootstrap.inc - the act of requiring the file loads a tonne of constants (including how many bytes in a kilobyte!) as well as the functions available in there.

The next code to get executed is bootstrap configuration, which does the following:

  1. Sees if sites.php is available and if there's anything in it.
  2. Based on sites.php and $_SERVER variables determines conf_path(), then includes settings.php based on the result
  3. The act of including settings.php populates $conf and some other globals like the database array.
  4. other bits and pieces we're not that concerned with here.

In this case, some global configuration for the drupal installation (which could have thousands of Drupal sites) may or may not be loaded before we access any context.

Then whether that's loaded or not, we use the context to determine which configuration to load. From there, all other context and configuration is determined.

Listen to me please, you will see that the main disagrement is really about how we name things.

Redefined

EclipseGc's picture

Sylvain,

I think the biggest problem we're having here is a communication issue, and most of that stems from the fact that you've been trying to redefine various words to meet your own pre-conceived notions about them for a while now. I appreciate that you've started to delineate between what we're saying and what you're meaning, but much grief could have been avoided if this had happened earlier.

Ultimately, as I said to pounard in IRC, I'm not really all that interested in the implementation details at the moment. That's not to say that they're unimportant (they're very important in fact) I'd just rather establish some consensus here, and that's all I've been attempting to do.

So I'll ask again, global configuration before or after a generic plugin system?

I see you introduced a

Sylvain Lecoy's picture

I see you introduced a concept which I don't know either, what are you calling "global configuration" exactly ?

Right, so this is what I've

EclipseGc's picture

Right, so this is what I've been driving at for so long now.

You seem to indicate here: http://groups.drupal.org/node/150149#comment-500459 that you basically agree with us. The problem is that, in your mind, the configuration of drupal as a whole (what I'm calling global configuration for the sake of argument) and the configuration of a plugin are separate things. This is not necessarily true.

But the drupal configuration (or configuration of plugin) doesn't have to be a part of the plugin configuration system...

There is a use case for plugins which simply work, i.e. they need no configuration, because they perform tasks SO simple, or so limited as to be easily represented by just a series of callbacks or methods. These sorts of plugins could easily be used for caching, session, and configuration swapping.

This is the critical flaw in most of the arguments I've seen coming from the new participants in this conversation. When it is suggested that "plugins" should depend on "configuration" the suggestion is that over-arching plugin system that we (the ctools people) want to see as the only plugin system would not be available until after a global configuration is bootstrapped and available, removing the ability for our plugin system to help this low level system. Thus if it wants a plugin utility (and I think it does from what I'm hearing) it would be a completely unique plugin architecture specific to configuration. This is sort of antithetical to our aims, we'd like to solve this problem, in its entirety during a single cycle. Not end up having this conversation YET AGAIN for D9.

Bottom line: Yet again, plugins DO NOT NEED CONFIGURATION. Some will, but they are MUCH higher level, these low level plugins can and should exists, rolling 1 or 2 separate systems for them seems really dumb to me when we could have a unified plugin architecture, and while I'm willing to make concessions on high level plugins having context and config forced down their throats (whether they need them or not) I'm wanting to see concessions to them not needing context and configuration in all plugins if we don't yet have them. This is reasonable, possible, plausible and useful. I still don't understand why there's so much resistance to us getting the most out of our utilities...

Eclipse

I'm sorry but it still

Sylvain Lecoy's picture

I'm sorry but it still unclear in my mind, why are you trying to remove the configuration from the bottom to put it after the plugin layer ?

I mean, if we take the example of caching: we basically have the choice between Database, APC, Memcache...

  1. I don't understand how you will be able to select one among the three if you don't have this information somewhere (global configuration)
  2. Maybe I'm wrong but the simple fact that we'll use a Database need some configuration somewhere (host, user, password). For APC the configuration is handled by php.ini but this is a configuration
  3. Your goal seems to be minimize amount of code to be loaded, but for Drupal, the only use case when there is no configuration loaded is when the website is about to be installed. This is a single-time use case and never after its being used anymore. Even for page caching delivery, settings.php is loaded. Don't you think that not having configuration in the bottom is a mistake ?
  4. To load this particular settings.php, you need to know in which context you are: for instance internal.mycompany.com or www.mycompany.com. Two differents configurations

So, let's assume the

donquixote's picture

So, let's assume the configuration is pluggable. Say, you can choose between the variables db table, a json-based filesystem approach and a yaml filesystem approach.

This might already use the mythical "plugin layer", or it could be a one-off system.

At some point we need to configure which plugin to use for the configuration. This setting can not live in the "configuration layer" itself, because that is not loaded yet.

So, no matter how the "plugin layer" is going to look like, there is one lesson to learn if we want the "configuration layer" to be pluggable: There has to be a second configuration layer, that comes before the actual "configuration layer" and that is not pluggable.
I'd say, this second layer is settings.php, and we are done with it.

So, we'd have
- A settings.php to configure the database, and to configure which plugin to use for the "configuration layer". This thing can only be web-written on site install, and then has to be maintained by someone with ssh / ftp access.
- A pluggable configuration layer, which is by default one or more json files. These files are written by Drupal (core, or a contrib plugin), they are subject to hook_update_n(), and any manipulation via ftp/ssh is at the site admin's own risk. This thing can be seen as yet another persistence layer, in addition to the database.
- Finally we still have the database. This is configured in settings.php, but it might not be loaded until later in the request. We put the definition in settings.php, because one of the "configuration plugins" might use the database.

Now, whether or not these "configuration plugins" use the new "plugin system" or something one-off, is a design question, that should be decided when the "plugin system" matures, and does not need to be answered here and now.

There is one question about this: Where do we store which modules are loaded?
1) Store it in the json files (the "configuration layer"). This brings a little problem: If "configuration plugins" are provided by modules, then we have no way to tell if the module providing a given plugin is enabled or not. So, it could be that we use the plugin to read the configuration, and then find out that the module providing the plugin is not available.
1a) One way out is that we always consider a module "enabled", if it is used in settings.php, and ignore conflicting enabled status in the json file.
1b) We do not let "configuration plugins" be defined by modules, but by something else. Then module enabled status can be safely stored in the json files.
2) Store module enabled status in settings.php? No way! Remember, the settings.php is not writable by Drupal.
3) Have yet another storage layer for module enabled status (replacing the "system" table), which is web-writable by Drupal, but not pluggable. Duh. I don't think we want that.


@Sylvain, your point 4.
At this point in time we can only use a "minimal subset" of context information. I think when we are talking "context", we want it to be aware of the current user etc, or don't we? So, not really the same "context" that we use to decide which settings.php to use.

Essentially the only thing I

EclipseGc's picture

Essentially the only thing I have to say to this is "YES". The "specifics" as it were are a little unimportant, but the essence of what was said here is exactly what we're arguing.

As for your questions/points about plugin usage:

I think that we'd have at least one "provider" of plugins called "core". Only core plugins would be loadable at this level? Something along those lines I could certainly see working, especially since I imagine we'd hard code the path to the plugin that we want to load. Yes a disabled module could provide "core" plugins as well, so we probably want to look at that, I know there have been discussions of a "plugins" directory in core for stand alone plugins (which... hypothetically would be the typical use case for these sorts of core systems). I don't want to gloss over the details because I think they're important, but I think the bottom line is that it is an implementation detail that should have a reasonable solution, so... we probably shouldn't worry about it right this second.

Thank you for this post, hopefully you put into words what I've apparently failed to communicate thus far. It is definitely very appreciated.

Eclipse

Ok, this would go in the

donquixote's picture

Ok, this would go in the direction of 1b), where we have plugins that are not part of any module.
I think this is ok and can work.
But, for easier project management, I think we want that one downloadable contrib package can contain a bootstrap level plugin (such as, a config reader), and a regular module. So we could say "drush dl yaml", and we would get both the config loader plugin and a module with some utility stuff. This also means, these things would all be stored in a sites/all/modules/contrib/projectname folder, not in a special sites/all/boot folder.
We might find that 1a) is easier to work with, where "everything is a module". In case of yaml, this would mean that the yaml loader boot module would be considered "enabled" because it is used in settings.php, and then the utility module would also be considered enabled, because the loader module depends on it.

Only core plugins would be loadable at this level?

The point of plugins is that we can install contrib or custom stuff and have it enhance or replace core functionality. So, if you say "Only core plugins", I hope you do not mean "only plugins shipped with core", but rather "only a special type of plugin that is designed to work very early in the php process, before we have other systems or data available". I think "core plugins" is a misleading term for that. I would prefer "boot plugins", as it indicates that these should work early in the process.

yes, so "core provider" i.e.

EclipseGc's picture

yes, so "core provider" i.e. core provides the structure for the plugin to hook into, not only core provides the plugins. Any module could provide a plugin where

$owner = 'core', $plugin_type = 'session'

just when we say "provider" we mean the module that is providing the structure into which the plugin plugs. Core (as it's been discussed thus far) would be a special case of "provider" specific to drupal itself (as opposed to say... system or blog, or whatever).

So, let's assume the

Sylvain Lecoy's picture

So, let's assume the configuration is pluggable.

Ok.

So, we'd have
- A settings.php to configure the database, and to configure which plugin to use for the "configuration layer". This thing can only be web-written on site install, and then has to be maintained by someone with ssh / ftp access.

Ok. If its written in the settings.php it doesn't need the plugin API does it ?

- A pluggable configuration layer, which is by default one or more json files. These files are written by Drupal (core, or a contrib plugin), they are subject to hook_update_n(), and any manipulation via ftp/ssh is at the site admin's own risk. This thing can be seen as yet another persistence layer, in addition to the database.

You then load the configuration layer, for me its like loading the configuration API, am I missing something ?

- Finally we still have the database. This is configured in settings.php, but it might not be loaded until later in the request. We put the definition in settings.php, because one of the "configuration plugins" might use the database.

You then feed plugins which needs database, I think its very similar to what I've described in my post bellow (http://groups.drupal.org/node/150149#comment-504724) ?

1) What about separate plugins and modules ? Without distinction of being enabled or not ? 1a seems an effective solution.
2) I agree that it is definitely not a good idea
3) No as well
4) The context should ba able to select a configuration, don't you think ? Let me explain:

Context #A
context->http_accept = application/json
context->http_session = 0939FU1S83CORcx
context->http_if_match = "ETag: 01930840"

You then give the configuration API the context object. The configuration API lookup for a hash of the context to see if there is a cached configuration in persistent layer for this context. KEY = context->getHash(). If not the context object get injected into plugins factories to get the correct implementation on-the-fly. The Rendering Abstract Factory will return a JSON plugin cause the factory knows to look for the http_accept key. The Session Abstract Factory will return the correct implementation of the session plugin, and so forth. The configuration API then save this configuration with the hash as key in the persistent layer, for further requests.

Does it make sense ? Do you think I understood clearly what you are saying ?

Ok. If its written in the

donquixote's picture

Ok. If its written in the settings.php it doesn't need the plugin API does it ?

For my taste, we do not need to use the "plugin API" at this level, we can simply have something one-off for bootstrap. The settings.php tells us which configuration loader to use, where to find the php for this loader (if necessary), and which parameters to give to the loader. Such as, where in the filesystem to find the json files. We do not need any "plugin API" for that. And we can discuss if the "loader plugin" can be part of a module or module project, or if it has to live somewhere else.

You then load the configuration layer, for me its like loading the configuration API, am I missing something ?

I'd rather say, a configuration "handler" (yeah that's also kind of a "plugin", just that you only ever need one instance at a time). Any config handler should implement the same interface, so the rest of Drupal does not need to care about which config handler we use.
We probably load a part of the config into memory at this time, but a lot of it can be lazy loaded when it is needed. I'd say, this is all up to the implementation of the config handler.
This is also the time when we know which modules are to be enabled, and where in the filesystem we find these modules.

4) The context should ba able to select a configuration, don't you think ? Let me explain:

I see the "configuration" as not much more but a storage engine. Similar to the database, but more light-weight (hopefully), and available earlier in the process. Somewhere in this storage we can have stuff that only applies to specific requests. But, this does in no way make the configuration layer depend on context. We can safely initialize the config layer, without knowing anything about the request (except those bits that decide which multisite we are in). Then at some later point, we can load settings from the config layer that are specific to application/json, or to some specific subdomain.

There are some things in the configuration, that should never depend on context. For instance, modules being enabled or not, and where to find them. Making this dependency would totally kill multiple contexts / requests in one php process. Which is still far away, but nevertheless something we should not intentionally move further away from.
Remember, most of a module should be lazy-loaded on demand, so there is no point in saying "for some requests we consider module x to be disabled".

Ok. If its written in the

donquixote's picture

Ok. If its written in the settings.php it doesn't need the plugin API does it ?

Btw, I think in other projects (Zend, symfony) you are supposed to write your own index.php, where you can choose which tools you want to use for bootstrap.

Ok. Let's take another

Sylvain Lecoy's picture

Ok. Let's take another approach.

@see: http://groups.drupal.org/node/150149#comment-505919

For sure using something that

sdboyer's picture

For sure using something that already exists will minimize risk but requirements are not fully met in CTools, and it shouldn't dictate design and system architecture.

Nobody is advocating this. NOBODY. Don't know how people have convinced themselves of this myth. We're doing what everyone else is doing - drawing on our experience the systems we know (or are learning about) and trying to figure out the best approach that considers all the factors.

The only thing that we're - or at least I - am lookin for is to not have the only people who've actually been doing a real form of plugins in Drupal for years being treated like red-headed stepchildren. And by that I mean...

Not having the context as input is like doing thing without really knowing why, it seems to me a blindly but easy approach, we should'nt be doing that. I know people will throw the golden hammer advocating using their growing product because they are proven to work well but we should expand our horizons and not being narrow-minded

Totally agree. Have since the beginning. Wrote up my research on Symfony2, and poked about at Kohana & Zend. And on balance, my take: there are a lot of great ideas from all over to be worked with. Lots and lots. And we should. But there's something they can NEVER give us that ctools can: real, functioning Drupal examples of what people have needed plugins for.

It doesn't matter if you don't like ctools' architecture. If you don't look past that to see the diversity of use cases that architecture has served and ensure your plans can accommodate them well, then...I think you're overestimating the importance of what "seems right to you."

I think you're overestimating

Sylvain Lecoy's picture

I think you're overestimating the importance of what "seems right to you."

That is personal but I admit this true, being so much frustrated by misconceived APIs and loosing so much time on it that even when I find a minor bug I bump it as critical in the drupal issue list. Sorry for overestimating my view point. Now, I'm trying to defend my thoughts, and pushing my ideas, as everyone is doing here. I like your approach of having a real background of plugins needing, and mixing with symfony2 to improve the existing is better than just migrate the CTools project.

better than just migrate the

merlinofchaos's picture

better than just migrate the CTools project

Every time someone says this, I will hear it as though a Fox News robot said Obama is socialist, and I will be unable to take anything you say seriously. I am sick of people repeating this false statement over and over again. Let. It. Go.

So listen to the robots. You

Sylvain Lecoy's picture

So listen to the robots. You are sick of me? I am sick of people who influence a . technical choice that widens his personal expertise and job marketability. But if we are sick together, at least we can listen each other.. :)

Please refrain from personal attacks

ksenzee's picture

Questioning Earl's motives does nothing to advance the issue at hand, and furthermore is against the Drupal code of conduct.

My statement is actually not

Sylvain Lecoy's picture

My statement is actually not so false and based on his sayings. He clearly said that if the choice wasn't the option he wanted, he would have to do it by himself anyway... Where is the Drupal code of response ?

So to get back to the subject: why the option #A is a danger ? And why when I ask Earl I get other people answering me. Its not that's its not nice answering me but basically I disagree with him so he has potentially the answer to my question.

Understand, ctools WILL

EclipseGc's picture

Understand, ctools WILL maintain a separate plugin architecture if this one doesn't work for us BECAUSE we already have a plugin architecture that not only we, but MANY other modules make use of. Therefore, we have a vested interest in this venture FAR beyond anyone else involved in this conversation (except Larry who's invested many years of his own time on this same topic), so understand that if this doesn't fit OUR use cases, we will be forced to maintain a separate plugin system.

As far as questioning Earl's motives, that's exceptionally bad form. That's about the only reasonable response I can give to your statements...

So you are clearly

Sylvain Lecoy's picture

So you are clearly contradicting sdboyer, who is telling me that no body dictate design and system architecture.

I am sorry to disrupt code of Drupal but I really think you are unfair with me, and not being honest by saying "this is for the community" while you have certain (and understandable) motivations about influencing this technical decision. Its just about that, nothing else. About Earl, I understand you are sick of people like me, but you have to understand that I got the right to answer, too.

Now that you told me the real motivations I am in a better position to listen to you. And that actually what I did, and I believe we actually all agree but terms we uses are just different in our mind. I will not rewrite what I said in this comment since you can read it here: http://groups.drupal.org/node/150149#comment-501109

About Earl, I understand you

merlinofchaos's picture

About Earl, I understand you are sick of people like me, but you have to understand that I got the right to answer, too.

No. Read what I said. I'm sick of a particular false statement that you, in particular, made and continue to make. Repeating it over and over has, as near as I can guess, made it true in your (and other people's minds). Stating that it's not true has not deterred you.

And now you take that as me personally attacking you, and attack me back. Your behavior follows the pattern of a troll, and there's no benefit to engaging it. So I won't.

"Context" should be late and local

donquixote's picture

As I understand, "Context" is the request- and user-specific stuff.
Imo this should be "late" and "local".

"late":
we don't need any of this to get our db connection, to read any configuration files, to decide which modules should be enabled (no we don't include any not needed module files yet, we just decide to consider them enabled or not), to prepare things to load on-demand. And that's pretty much all we should do in a bootstrap.

"local":
I would appreciate if we could keep this request-specific stuff out of global scope. No more $_GET['q'], and no Context::get('user').

I would like the following (simplified) to be possible from anywhere in Drupal:

<?php
$request1
= array('user' => $u1, 'path' => $p1, 'POST' => $post1, ...);
$request2 = array('user' => $u2, 'path' => $p2, 'POST' => $post2, ...);
$context1 = drupal_build_context($request1);
$context2 = drupal_build_context($request2);
$theme1 = drupal_find_theme($request1);
$theme2 = drupal_find_theme($request2);
$response1 = drupal_serve_request($request1);
$response2 = drupal_serve_request($request2);
?>

This would allow different contexts, and different requests, to coexist in one php process, and not interact with each other on global state.
Just imagine how nice this would be, if only for testing.
(at some later point we might go even further, and do the same with two different db connections etc)

This a very admirable model.

moshe weitzman's picture

This a very admirable model. Might take us a while to get there, but I think this is a wonderful target. PHP does not make this easy. Have to kill conditional include of files, $GLOBALS, pesky statics, etc.

yeah, all those things we

donquixote's picture

yeah, all those things we should get rid of anyway, some day in the future.

Pluggable Configuration

mfer's picture

One of the things I see a need for is a pluggable configuration system. Configuration shouldn't be hard coded for JSON. Configuration could be in native PHP (which will be great for opcode caches), XML, YAML, or INI files. Different formats for configuration files are in Zend Framework, Symfony2, and other systems.

This is going to be important for out interaction with other systems. Unless we want to have one off config systems for working with other systems (like Solr).

Does this mean, you want to

donquixote's picture

Does this mean, you want to (i) have a choice for how you want to store the drupal configuration / variables ?
Or do you just want (ii) different configuration systems to be available for other purposes, while the drupal vars are stored in JSON anyway?

For (i), we would still need one place to start, a file that tells us which storage system to use, or at least gives a hint. And then, if these storage plugins are provided by modules, we would need some knowledge of which of these early bootstrap modules are enabled or not.

Also, if you uninstall the only module that can read your config, you are screwed.

I apologize

Crell's picture

It seems I shouldn't have posted this right before getting on an airplane without more advanced context (no pun intended). And I'm regretting even summarizing the A and B possibilities that have been discussed. So let me try to clarify something here:

1) This thread is not nor should it be a referendum on ctools or its approach to plugins.

2) This thread is not should it be an Earl vs. Larry cage match. (I'm seeing a great deal of ctools vs. Larry language above, which is extremely counter productive.)

3) There is not nor should there be any "versus" here at all!

Campaigning for Option A or Option B is not productive here. That is not going to accomplish anything. What I was hoping for is that by trying to lay out the situation for a broader audience we could come up with an Option C. That is still my goal.

To that end... I am going to ask that the people who have already been debating this question for years or have already been extremely active in the WSCCI group to date please shut the hell up and give someone else a chance to talk.

We are not going to accomplish anything by having the same people having the same debate again in yet another venue. That is a waste of everyone's time. We need new voices involved that can think outside of the box we've been using, and preferably with experience on systems outside of Drupal. (There have been a few of those posts above, which is great.)

Unfortunately so far I see 50+ comments that, wearing my Initiative Owner hat, have accomplished nothing.

Very well, I'm done. I will

merlinofchaos's picture

Very well, I'm done. I will butt out of the conversation.

When my experience counts for nothing, and I apparently can't think outside my own box, and it's okay for you to pick a side and 'campaign' for it, but not for people who disagree with you, there is no conversation to be had. Good luck with your initiative.

Dude, great ideas come from

Sylvain Lecoy's picture

Dude, great ideas come from great debates. I am listening to your examples and your thoughts, and I m sure Crell too.

You are right, it was a

Crell's picture

You are right, it was a mistake for me to comment at the start of the thread in the first place, and I apologize for that.

The point is not that your experience counts for nothing; it's that I am actively trying to get the experience of more people than have been commenting so far... and having the same people (including me) commenting over and over keeps new voices and new perspectives from having a chance to be heard.

whose perspective are you looking for?

ksenzee's picture

I'm not sure whether to comment on this thread, because I don't know whose perspective you're looking for. I assume since you're posting on g.d.o that you want to hear from people in the Drupal community. Are you looking specifically for people who have needed plugins but haven't tried ctools plugins as a solution? Or are you looking for people who haven't had any need for plugins in Drupal?

Logic problems

Crell's picture

Ideally, people who can approach the difficult logic problem here with a fresh set of eyes. Experience with both Drupal and some other 3rd party system would be ideal. It's not plugins per se that are the issue, it's the interaction between these three systems, only one of which is plugins. No one so far as talked about needing to configure how to find your configuration. :-)

Yup, because configuration

pounard's picture

Yup, because configuration has not been well defined, and can be so much different things, from the pragmatic array, to the variables, going through contextes themselves that can be. While this can be everything, nobody will argue the need to have at least something.

Pierre.

hm?

catch's picture

My eyes may not be fresh, but that's exactly what I discussed in this comment - http://groups.drupal.org/node/150149#comment-499354 - however it hasn't had any responses at all. Would be nice to see some (especially on things like sites.php which is configuration that by definition is loaded before we even know which site we're in) instead of all this meta-discussion.

I actually quoted you here:

Something has been achieved

ronald_istos's picture

That last statement is kind of harsh Crell.

I think that what has been achieved is:

  1. That some people are arriving (albeit via heated debate) to the fact that there is an Option C - which is not an either or type of solution.

  2. We have some examples of how different people envision this to develop.

  3. We have the very valuable input of CTools people that have been dealing with this for a long time.

As far as the out-of-the-box, etc - Well, we can debate endlessly about which approach is better and given that there are competent people thinking about this we will probably come up with reasonable solutions. I really don't think there is a lot outside voices can provide beyond stating overarching principles and approaches (keep it simple, be realistic, don't forget about performance, make sure you are not creating non-existent dependencies, etc).

What outside voices are lacking is actual real, practical, when the s**t hits the fan experience with core and/or other similar such systems in Drupal (i.e. CTools). So one can argue that since the broad approach has been largely sketched out (flexible config, powerful context, supported by one or more plugin systems) what we need right now are the voices of experience to guide the exact implementation.

Unfortunately so far I see

sdboyer's picture

Unfortunately so far I see 50+ comments that, wearing my Initiative Owner hat, have accomplished nothing.

If you can't see any value in the comments here so far, you might want to consider the possibility that you're listening incorrectly. Or that what (or who?) you're looking for out of this discussion is unrealistic.

I really think this

catch's picture

I really think this discussion needs better definition of what the terms are, and what the use-cases are (merlin's examples help to define this). And when discussing interdependencies we need think very carefully about what the absolute minimum amount of code and data is required before you have access to the rest. When I have asked Larry what his plans for minimum bootstrap are, he said "I want to get rid of bootstrap and start from scratch" - this is a nice desire maybe, but it isn't a plan.

The hardest use cases are likely to be the very lowest level (cache system, page caching), and the high level (image derivatives, language settings etc.).

The dependencies for higher level stuff don't matter so much - you can already assume you have a lot available by the time you get to them. Allowing the high level stuff to use the same patterns as the low level stuff is a good goal though, but it shouldn't add too much weight for requests which can get by without having everything available either. I do think there's the potential for both of these to be done successfully.

For low level stuff it is very important what exactly the dependency chain looks like.

Something that we broke, albeit accidentally, several times during the Drupal 7 release cycle (and it's still broken in 7.0 and only somewhat fixed in 7.x-dev) is page caching performance. Any change to bootstrap has a good chance of breaking it again.

For example, let's look at index.php, drupal_bootstrap(DRUPAL_BOOTSTRAP_CONFIGURATION); and drupal_bootstrap(DRUPAL_BOOTSTRAP_PAGE_CACHE);

index.php defines DRUPAL_ROOT to get_cwd() then does a require_once of includes/bootstrap.inc - the act of requiring the file loads a tonne of constants (including how many bytes in a kilobyte!) as well as the functions available in there.

The next code to get executed is bootstrap configuration, which does the following:

  1. Sees if sites.php is available and if there's anything in it.
  2. Based on sites.php and $_SERVER variables determines conf_path(), then includes settings.php based on the result
  3. The act of including settings.php populates $conf and some other globals like the database array.
  4. other bits and pieces we're not that concerned with here.

In this case, some global configuration for the drupal installation (which could have thousands of Drupal sites) may or may not be loaded before we access any context.

Then whether that's loaded or not, we use the context to determine which configuration to load. From there, all other context and configuration is determined.

So, are we going to require that the actual context system is available, just to access a couple of superglobal variables, or do we allow those to be accessed directly at this very early stage but never directly again?

Then bootsrap page cache does this:
Checks the database for blocked IP addresses - that is just silly and i have a patch to move it out - http://drupal.org/node/1161486

Initializes the cache system (via configuration from $conf)

Includes a hard-coded, very hacky (but very performant when it's not broken) workflow for handling the page cache - which allows it to serve an http response without a full bootstrap. That also needs to load some basic context into it - like do we have a session or not.

The ideal thing for page caching to make it not a hack would be this:
- you get a request that is content type text/html and GET
- that gets routed to the thing that handles HTML GET requests (probably a plugin)
- that plugin contains the current basic page cache logic (or routes to yet another plugin) that can check if there's a session and if there page is cached, if it can serve from cache it does, if not it starts the page building process.
- all this happens quicker than the current hacks allow it to now.

This is not such a huge ask. With normal page caching we already have to load bootstrap modules and the full variables cache, just to find out if a page is cacheable or not. We already have hacks in settings.php with $conf['page_cache_without_database'] etc. that allow you to use disk based configuration only (settings.php itself) to determine whether to serve a cached page or not. So if done carefully, it could be closer to aggressive caching performance, not just equivalent.

If we get this right, every request type has a chance to be as fast as page caching is now (or rather, as fast as it was in D6) - some might never be, but they will start on an equal footing. Additionally the code path will be split extremely early on.

If page caching performance plummets, or we have to leave it in as a one-off hack to avoid that happening, then something during the whole process went wrong.

I know everyone wants this or something like it, and it is part of the reason for the context/services initiative at all, but without some basic use cases (like the features that Drupal core has now) it is not surprising that the discussion is circular.

Another thing we need to balance is the need to be able to access a subset of configuration early (to avoid loading huge arrays and objects into memory that we then do nothing with), vs. loading information twice. Right now, in Drupal 6, bootstrap modules are loaded during hook_boot(), then again as part of the list of all enabled modules on full uncached page requests - that's a waste for those uncached requests - they could just get the full module list at the beginning and skip the round trip. Then, because module_list() is buggy, not only are the same modules loaded for hook_boot() and full, but the list is loaded from the database yet again for hook_exit() instead of re-using the original list we fetched at the beginning of the request. There are similar problems with variable_get() being used for thing that can only ever be set in settings.php, vs. those loaded from variable_init() - if we're going to have multiple-layers of configuration available (and I hope we do compared to loading thousands of items into a big global) we have an opportunity to learn from the mess that is currently there.

<blockquote>The

pounard's picture

The dependencies for higher level stuff don't matter so much - you can already assume you have a lot available by the time you get to them. Allowing the high level stuff to use the same patterns as the low level stuff is a good goal though, but it shouldn't add too much weight for requests which can get by without having everything available either. I do think there's the potential for both of these to be done successfully.

Totally agree here. Nevertheless I'm not afraid of performances when loading some interfaces and basic implementations. You can init a full and complex API without really loading it with a good autoloader. Once high level is booted up you can then initilialize using configuration or database some more high level API, and use more specific component that are dependent over a lot of those low level interfaces.

That is also one the benefit of using OO code for this, and a good autoloader. You don't have to worry about loading too much anymore, if your design is not broken, you will only load the bare minimum, and even better, the interpreter will do it for you, you won't have to worry about.

The ideal thing for page caching to make it not a hack would be this:
- you get a request that is content type text/html and GET
- that gets routed to the thing that handles HTML GET requests (probably a plugin)
- that plugin contains the current basic page cache logic (or routes to yet another plugin) that can check if there's a session and if there page is cached, if it can serve from cache it does, if not it starts the page building process.
- all this happens quicker than the current hacks allow it to now.

This is basically what is being discussed in one or two other WSCCI posts. This isn't within this post scope.

Another thing we need to balance is the need to be able to access a subset of configuration early (to avoid loading huge arrays and objects into memory that we then do nothing with), vs. loading information twice. Right now, in Drupal 6, bootstrap modules are loaded during hook_boot(), then again as part of the list of all enabled modules on full uncached page requests - that's a waste for those uncached requests - they could just get the full module list at the beginning and skip the round trip. Then, because module_list() is buggy, not only are the same modules loaded for hook_boot() and full, but the list is loaded from the database yet again for hook_exit() instead of re-using the original list we fetched at the beginning of the request. There are similar problems with variable_get() being used for thing that can only ever be set in settings.php, vs. those loaded from variable_init() - if we're going to have multiple-layers of configuration available (and I hope we do compared to loading thousands of items into a big global) we have an opportunity to learn from the mess that is currently there.

The more you will spend time on attempting to load the bare minimum, the more time you will loose. This seems like a paradox, but this implies the core to implement a complex "meta code file handling" while the PHP interpreter and OPCode caches basically already have some optimizations for this (the later has a lot).

Design should remain simple and this is complex. This is pure micro-optimization, and IMHO, doesn't belong to this structural discussion.

Pierre.

The page caching stuff is

catch's picture

The page caching stuff is entirely within scope (at least, within the scope of Larry's original post, not necessarily the argument about plugins specifically).

I don't think people are sufficiently differentiating between the plugin/context/configuration systems and what context or configuration will be available at any one time. Loading more or less all our possible configuration into memory as we do now for every request regardless of what it needs to serve is killing us both from page response time and memory usage. This is well documented in the 'memory' tag on Drupal core issues.

Design should remain simple and this is complex. This is pure micro-optimization, and IMHO, doesn't belong to this structural discussion.

Please read all of http://drupal.org/node/1064212 then come back and say this again. If you guys ignore these points now and fuck up Drupal performance by implementing something pure and simple but completely slow and unscalable, I will not be stepping into help after the fact.

Plugins implement certain interfaces

torotil's picture

What I am missing in this discussion and in the definitions is that there is no mention of „interfaces“. IMHO plugins are pieces of code that implement a certain interface. It provides a certain functionality through this interface. The interface itself has to be tailored to what functionality the plugins should provide. So essentially plugins separate what kind of functionality should be provided (interfaces) from how that functionality is provided (plugins). So the interface is specific to the functionality (ie. a storage plugin surely has a diffent interface than a interface related plugin).
What I miss here is a discussion about how the interfaces should look like and where they should be defined.

I'd also prefer the plugins chosen in some more basic way, like an extra plugins.config.php. The plugins for storage, configuration, request handling … are not likely to be exchanged very often during the life-cycle of one drupal site. And if they are exchanged (that means change the innermost behavior of a site) chances are that it needs more than just a little work anyhow and editing this a php-config will be the least of your problems.

Interface is the word that

pounard's picture

Interface is the word that shortcut the design by contract well-known design pattern. Design by contract can be implement in many ways (Plone guys implemented it as an API that heavily rely on runtime checks), while other Java-looking OO languages (PHP is one of them, C#, even C++ for what it worth) implement the "Design by contract" pattern directly into the language itself, by using the keyword interface.
Basically, an language keyword based interface is not really flexible, but it highlight design. While it's not flexible (you implement flexibility in the design itself and problem solved by the way) it has proven great performances because contract is proven and verified at compile time and not at runtime. Using a good OPCode cache really speeds up a lot this kind of interface based designed code.
While this is not totally true for PHP, because it has dynamic runtime type checks, this still remains faster because it's an optimized part of the interpreter (that will always be faster than a reinventing the wheel implementation such as the actual CTools implementation).

This is already too much said, I wont give you the study time you missed when you learnt IT development, it was invented back in the 80's (may be even before) and now is popular enough from 10 to 15 years so that any developer in the world, even if he doesn't know OO code should probably already have heard of the design by contract pattern.

Pierre.

?

torotil's picture

Yeah right … In what regard is this an answer to my post?
Thanks for bringing in the "design by contract" keyword, which is not exactly what I had in mind. It is already overloaded with how things are "usually implemented".

We still don't know whether or not we use PHPs interface construct for plugins or some other construct. Or did I miss the discussion where this is fixed?

Sorry, totally answered

pounard's picture

Sorry, totally answered aside. The "Design by contract" seems good enough to define interfaces, whatever is the meaning of it (high level API or lowl level interface keyword doesn't really matter).

The fact is there are both business logic interface (where core shouldn't really interact) and low level API interfaces (should plugins accept a configuration object or not). So basically, plugins won't implement only one interface, but a set of interfaces, the first set for API consistency and plugin management API, the second set, business related and defined by an higher level module that cares about business stuff.

Don't know if this answers to the question.

Pierre.

pounard's picture

Ok I admit I am a good factor of this post failure. That's why I'm gonna do a useful post for once.

I think that most people probably are confused by the new vocabulary. The actual WSCCI definitions page is really great for starting understanding why we are here. The problem is that I didn't see the equivalent for the new configuration layer, which is probably one of the sources of confusion.

When I talked Greg Dunlap about the Zend_Config API, it was originally about having a mapping layer interface for abitrary configuration needs, not about bootstrapping what exists as known as the variable system. I see, but I'm not quite sure Greg got it, that most people doesn't really understand it.

What I understood when Crell spoke the first time he got here speaking about context, configuration objects and others revisited concepts, is that plugins are what I call components (the words here pretty much are the same, except I like to remove the history side of the plugin word through the Drupal project), and they probably would need at some point to be set-up before usage. Set-up doesn't means a full configuration layer (neither means variables by the way) but somehow, he finally got to the point that (I suppose), most components would be dynamically spawn depending on the context: in order to handle all the same way (common API to rule them all) we would need to specify and design a common set-up layer API.

Here, when I'm speaking about context don't get me wrong once again, not speaking about the historical meaning of this particular word into the Drupal project's history, I'm speaking about arbitrary contextual information derivated from a lot of things. Basically, context is a set of key/value pairs, that can be structured or not.

What is, for me, the mapping layer? I'm here speaking about a really low-level API. This mapping layer only define a new type of object which is the configuration object, more than an object, an interface. Basically we could define the fact that our mapping layer here is any object that derivates from the ArrayAccess and Iterator interfaces. This only means that the mapping layer pretty much defines itself by its design, we don't care about the implementation here. If we have this mapping layer, why wouldn't we also provide merge, override and export common method signature throughout all the objects so we can potentially use them as an import/export layer? This what I'd like myself, and this what would make it a real mapping layer.

Once again, that defined, my mapping layer and the configuration objects concepts are now merged, and design here is pretty clear. Remember I still not talked about variables.

When Crell tells that plugins needs to be configured, I understand it that way: If I have, somehow, to spawn dynamically a plugin without knowing the business logic, I need at least it to have a known method for me to give them a configuration object that have been spawn by the contextual runtime..

When he speaks about the configuration layer, the object set-up, to give the context, he only means give a full shitload of data inside some kind of arbitrary structured object, and let the plugin using it or not. What the plugin layer needs is only a setOption(My_Configuration_Interface $options) method, and that's pretty much it! The plugin can derivate from a neutral implementation that just implement a void method for this, none of my businesss! None of Crell's business! And most important, nothing more than a single method, and absolutely no performance impact.

All I need to bootstrap to have a fully featured mapping layer is an array-based implementation of my configuration object interface that is typically independent code. It doesn't even need Drupal to run! He just need PHP 5.2 and no more.

So, what I'm saying here, don't confuse the configuration API and the variables. If I implement the configuration API, or what I prefer to call the mapping layer base layer, I can already use for plugins, even if I don't have database, and even if I don't have variables.

Variables, for the sake of consistency across Drupal's framework should probably be a specific implementation of the mapping layer, that I can bootstrap anytime later.

I totally agree with Crell, plugins needs a configuration common interface even if specific plugins implementation remains neutral and doesn't use it. This common interface doesn't need, in anyway, to have the variable system bootstrapped, at anytime!

That said, the original bootstrap problem is resolved, this because once you bootstrapped the mapping layer that depends on nothing, you can bootstrap the plugin system, that only relies on the mapping layer by design, you can then start to spawn context objects, that are basically mapping objects but also can be plugins! Once this done, you can use it for cache backends, and for variable system backend.

If you still don't want the mapping layer or configuration API (call it as you want) inside the plugin system design, then you have absolutely not understood what it offers. It offers unlimited import/export abilities, for everything, only by adding an simple interface and one or two framework independent implementations, and it also offers a pragmatic way to instanciate any specific businness logic oriented plugin using bits of configuration (that the plugin might or might not use) at the lowest cost possible over performances, with one and only one common API that pretty much work for every god dawn use case you can imagine.

And for the sake of fucking not reinventing the wheel, I gave you two possible implementations of this mapping layer, one from ZF known as the Zend_Config component, and one other that I did myself that integrates well with Drupal, since it's a module I use almost every day.

Pierre.

Configuration VS. Settings

Sylvain Lecoy's picture

To me, a configuration is how plugins will be resolved at runtime. It rely on a mapping layer.

Settings are the actual configuration of the plugin, as we know as exportable, but I don't like using the word configuration as it conflict with the former definition.

Both configuration and settings can use the same key/value store, but there are conceptually different, and that's lead to a lot of misunderstanding between developers.

Pretty much what I described.

pounard's picture

Pretty much what I described. But, plugins won't be resolved at runtime by configuration, but more by a discovery system (what patches and CTools actually does).

Mapping layer is a lower level API for common arbitrary configuration data structure management, whatever they come from configuration or pragmatically set arrays at instanciation time or context data.

Variable system is just a specific implementation of the mapping layer, accessible from a singleton somewhere, that acts like a common environment or site wide related variables that anyone can use.

Pierre.

Yes I had this in mind as

Sylvain Lecoy's picture

Yes I had this in mind as well, sorry. To clarify by giving a sample:

Configuration #1:
- OAuth PECL plugin
- MySQL plugin
- HTML Rendering plugin

Configuration #2:
- OAuth PHP Lib plugin
- MySQL plugin
- JSON Rendering plugin

Configuration #3:
- OAuth PHP Lib plugin
- MongoDB plugin
- AMF3.0 Rendering plugin

Configuration #4:
- OAuth Stub plugin (testing)
- Database Mock plugin (testing)
- Internal Browser Rendering plugin (testing)

Settings for OAuth based plugin:
- consumer_key: 45DJE6C8Z9iXOKZn_O9
- consumer_secret: 5DF3KoicieRCD-7ek

Settings for MySQL plugin will be different than for MongoDB plugin.

Rendering plugins can have shared settings and context driven settings.

Then we see that clearly a configuration is just a set of plugins which is proven to work together, either created on-the-fly or by a dev, or sys-admin. The context then gives enough information to select a configuration to appropriately respond to a request. And yes we can see that as magic injection like merlin said, but its actually well defined, and process is well controlled. But that's explain the needing of a Configuration layer as a base. The settings layer, however, can be build on top with a factory (like said here: http://groups.drupal.org/node/150149#comment-499099). I think we mainly agree on the same ideas, the debates, in my opinion, come from this difference between configuration and settings.

What I call Settings is called Plugin Configuration in the Definitions Page:

Plugin configuration is user- or admin-supplied data that affects the behavior of a plugin. It is context-free, that is, it does not change with the context information but complements it. A plugin's behavior may be affected by context, by configuration, or both. In the present architectural plan plugin configuration will be implemented as an object that is passed to a Plugin. It will be persisted using whatever mechanism is developed by the Configuration Initiative

What I call Configuration is a set of Mapping definitions for a given Context:

A configuration represents several mapping (called bindings). These mappings are defined between interfaces of components (required or provided) and roles of connectors (input or output).

In Drupal we can't really speak about connectors between components but typically the MySQL plugin is a connector between the Database Abstraction Layer and the component MySQL Server. The role of this connector is a one-way only (Database can't call Drupal).

Strictly speaking, Drupal plugins as we see them are not components, but more connectors. But as connectors can be components, this is not a problem. If we see Drupal as a component, needing another component: Database Abstraction Layer, let's call it Persistent Layer (since with some caching system it does not use database anymore), an OAuth component, and a Rendering component. Each components can be bound to a connector within a configuration. In the configuration #1, OAuth component is bound to the PECL OAuth Connector (which is really purely a connector between the OAuth provided interface and the Native C implementation). Persistent Layer is bound to the MySQL Connector, Rendering component is bound to HTML connector, and the whole configuration contains these components, connectors and bindings.

That's why, in my opinion, the configuration need to be the bottom of the stack. Without configuration (mapping layer, or whatever), you don't know what plugin to load. Without context, you don't know which configuration to load. And this is very important to understand that this doesn't mean configuration and context should be injected by construction in the plugins, which is a terrible mistake, as CTools folks underlined.

Zend_Config

donquixote's picture

I think there is one problem with Zend_Config: It is read-only. (afaik)
The site builder / developer is supposed to get into the file and customize the values manually.
No hook_update_n().
See http://groups.drupal.org/node/149769#comment-498574

Zend_Config is absolutely not

pounard's picture

Zend_Config is absolutely not readonly, it's a parameter. Some implementations of it maybe per default, not all by the way.

Pierre.

Zend_Config_Writer

mfer's picture

There is a second set of classes for writing in Zend. See the documentation for 1.x. These are also in the 2.x branch. Since reading can happen on every page load (or almost all of them) and writing is a much rarer task; separating the read and write code from each other makes sense for a lot of reasons.

Totally, but it doesn't mean

pounard's picture

Totally, but it doesn't mean they really have to be separated in code. I think that some configuration objects (such as variables object) may want to implement an active record pattern instead of relaying on a custom writer object that does it only once a hit.

Pierre.

Nice, good to know :)

donquixote's picture

Nice, good to know :)

EDIT:
And I have to admit, this was a short-sighted comment of mine. Even if the reader plugin is read-only, or if Zend does not use the write option, this does not mean there cannot be a way to machine-write the data on module update.

I'm auto replying my huge

pounard's picture

I'm auto replying my huge post because I uploaded an UML class diagram there.

Where Earl attempts of mock code seems efficient for a lot of people, I prefer speaking with well known diagram formalism, my memory is sensible to pictures. While mock code sample can make people see what would be some use cases of the API, a fully feature class diagram highlight the dependencies and focus on the initial design without worrying about implementation whatsoever.

Pierre.

We've talked this through on

merlinofchaos's picture

We've talked this through on IRC, but I would like to point out that your diagram does not actually support option A, which proposes 3 different plugin systems (1 generalized one and then 2 one-off systems that #1 is dependent upon).

I haven't really looked that closely at the diagram (just the parts that really concern this conversation) but I will say that it has a lot of interfaces that on the surface are superfluous, but maybe just need to be justified.

Most interfaces are here to

pounard's picture

Most interfaces are here to highlight the design. Using interfaces allow to unify API signature letting the services (plugins, configuration, etc) providers override anything while keeping compatibility with the core API. There are not that much IMHO, but don't hesitate to argue here I'm open to suggestions.

Pierre.

For fun I did a fully

pounard's picture

For fun I did a fully functionnal implementation there, it's about 300 lines of code, documentation included.
And ported this older code working using it.

Pierre.

Goodbye "generic Plugin system" - for now.

donquixote's picture

So.. to get this debate somewhere else, what about
- We say goodbye to the idea of a generic plugin system, for now
- We produce dedicated "plugin systems" for database engine, filesystem-based configuration storage engine, and other things we want to make pluggable. All of this taylored to the specific use case, and to what information we have available at this time in the process. Some of them can use the modules system, others maybe not.
- We look for similarities, and things-to-learn, and then decide if we want to design a more generic plugin system.
- Then we port those new systems to the new generic plugin systems, where it makes sense.

As Merlin pointed out, the db layer and other things can already considered to be "plugin systems" in theory. But their architecture can be improved to be more like what we imagine a plugin to be like. Then we can talk again.


I really would like to

pounard's picture

I really would like to differentiate, for the sake of clarity of mind (and ensure newcomers will understand) what we could call plugin:

  1. Plugins (yes, sound stupid said like that), probably business logic oriented plugins, those high level components this plugin and context thread probably points out the most;

  2. Environment related backend, basically plugins, but somehow "slotted" (cache backends are being used as singletons for each bin). This is where we split and where I think that variable backend or cache backend ARE NOT high level plugins. Doesn't mean they should not accept configuration, but it clearly means that the choice of these should never be made at runtime, and are purely hardcoded configuration driven. The sysadmin chooses the cache backend depending on the underlaying architecture and the developer should never made this choice and even worst override it. Variable backend is the same, I understood pretty well that, for example, variable sharing among multiple frontend is purely an physical architectural and choice does not belong to Drupal framework.

The first category needs a high level configuration layer, and would probably access variables, while the second doesn't need variables, it only need the settings.php hardcoded variables (to fit with the actual way of doing).

EDIT: This is why separating what you call plugins and what I call backends really clarify the design and definitely allow a generic plugin handling API.

Pierre.

Right now we give up a lot of

catch's picture

Right now we give up a lot of things by having variable_get() work for both things that should only ever be in settings.php (like your example of memcache config), and things that never should be (cache clear timestamps). If we could actually make this a hard separation and stick to it, then it would make far less of a mess of interdependencies.

I know hejrocker has said he'd like us not to have to have the db settings in settings.php, while I'm not convinced either way on that, database configuration is a lot higher level than the settings for the configuration backend, or the cache backend - so it wouldn't be impossible to move that out of settings.php, while keeping it for the rest.

Follow the Butcher

moshe weitzman's picture

Good gosh. All this astronaut engineering does not bode well for Drupal's future. Is this what future development discussions look like? Matt Butcher warned us of this path ...

Why does Drupal have

pounard's picture

Why does Drupal have thousands of modules while other CMSes -- many older than Drupal -- have far less? The reason is as clear as day. Drupal modules are simple to develop. And this is not just because hooks are easy to use, but because the API has long been focused on making hard things easy.

Another reason why this post is already obsolete is that since D7, core is complex, really complex. It's now being hard to develop with Drupal if you don't masterize (there is no word big enough for this) project's history (at least since Drupal 5).

Plus, this thread is talking about a really simple design. If you are so afraid of OOP code, just take a look at ZF for instance, Symphony, new Typo3 engine, Java Beans, Apache Java modules, Magento, I could quote here many other pieces of software.

A good design is always much more readable (procedural or OOP doesn't much change this statement) if it has been well defined. That's exactly what is being done here.

Pierre.

but because the API has long

Sylvain Lecoy's picture

but because the API has long been focused on making hard things easy.

I would rather say, the API is focused on making easy things hard.

Module enabled status - where to store?

donquixote's picture

And here is a relevant question about the planned filesystem-based configuration storage layer.
Where do we store which modules are enabled? In this new fancy storage layer? Or some place earlier than that?

If the storage layer is to be pluggable, then will these plugins depend on modules being enabled or not? Or will these plugins be provided by something other than modules? (such as drush, which is not a module in the sense that it could be enabled or disabled).

These "other things", that are neither module nor theme nor install profile, could be shipped in the same way as modules, and stored in the modules folder (such as drush), or we could have a dedicated sites/all/boot or use the sites/all/libraries. Then we'd use the settings.php to configure which of these plugins should be used for configuration storage.

Only then can we find out which modules are enabled or not, by starting the configuration storage reader plugin, and looking into the configuration that replaces what is now the "system" table.


In the common OOP frameworks like symfony or Zend, it is common practice that the site developer will "hack" the index.php, and decide which tools should be used for configuration and bootstrap. We will rather use settings.php, I would say..

Option C:

EclipseGc's picture

OK, so I've tried to say this a few times, this is my last attempt:

Option C:
Option C is essentially 2&3&4&5... these 4 items are NOT mutually exclusive (I believe Larry actually pointed this out already).

Let's Analyze:

3. Implement a simpler, more streamlined pluggable mechanism for configuration other than plugins.
* Pros: Eliminates the dependency on plugins. Allows for a finely-tailored light-weight system in a very performance-sensitive area.
* Cons: We're back to having multiple pluggable mechanisms, even if we only plan on the one one-off.

Great, so... the plugin system ends up with a base plugin class that is "irreducibly complex" in that it's JUST the plugin system, it accepts no configuration, no context. This is the most basic plugin system core possess, any process happening before the configuration system can make use of this plugin system as it's generic enough to handle any of these situations, and it is NOT just in existence for configuration. We now have our base upon which all else it built, configuration is still plugable, but the mechanism isn't foreign to the rest of the plugin system because HEY, this is what everything else is based on. Pretty sure we can maintain the pros, and remove the cons.

2. Implement a simpler, more streamlined pluggable mechanism for context handlers other than plugins.
* Pros: Eliminates the dependency on plugins. Allows for a finely-tailored light-weight system in a very performance-sensitive area.
* Cons: We're back to having multiple pluggable mechanisms, even if we only plan on the one one-off.

Now that configuration is available, we extend the plugin base class to consume configuration. At this point we can now create plugins that are configuration-less, or that actively utilize configuration. This entire system is available to the context system for its use, and again, if there's something else that happens after configuration, but before context, we have a generic plugin tool available should it need it.

Finally, once context IS loaded, we extend the previous class, and consume both context and configuration, and all plugins from that point forward have a consistent mechanism. This also means that, should we need it, even at these higher levels, we could have config/context-less plugins as necessary.

Pros: We've not re-invented the plugin utility at various random spots. We've implemented a sound, core approved methodology for extending the plugin system (something I'm sure contrib will come up with some additional need for at some point), We've not re-invented the plugin system for two subsystems (or any other unforeseeable subsystems that come up with a need and no solution), and we've HOPEFULLY made both sides in this argument happy.

Cons: This may not fit everyone's ideological perfection, for my part, I've always defined what a plugin type needs when I defined the plugin type, and there have been plenty of situations where I needed neither context nor configuration. While it seems cleaner to me to not even have to deal with it if I don't want to, I'm willing to accept a compromise like what I've outlined above. Hopefully we can find some common ground here?

Eclipse

When you define something "by

pounard's picture

When you define something "by design" such as the abitility to use a configuration, doesn't mean you really need to use it. I mean, I agree a lot of plugins are pretty much stateless or contextless, that's not the point I'd argue (probably a lot will be anyway). What I mean by including the ability for plugins to accept it (even if it's a dummy/void method) is the fact that the toolbox is there for everyone, even if you don't use it.

It make pretty much sense, and allow callers to pragmatically instanciate objects by passing through configuration objects and chain method call without having to test the plugin(s) real nature behind (this kind of runtime checks would probably cost more in the long term than creating the void method on plugin base class).

Where I want to go to is I pretty much agree with you, why not having a granularity in what is a plugin. That's probably a good concept, but it makes harder the rest of the API because anywhere you'd use plugins in a generic manner, you'd need to test their nature and derivate your own generic code. The whole arround API code would probably be larger and less comprehensible, while having a foo void method implemented only once at the top level class definition is so much less code in order to achieve the exact same goal.

I see this as a tool that you totally have the right to ignore, but nevetheless a useful tool for other and less code for the overall framework.

And in the solution I describe, I'm not talking about site configuration or context but just about a simple method that accepts a configuration object that can totally ignore it. Which is way more lightweight and provide minimal coupling (it couples the plugin with one, and only one interface, which is ConfigurationObjectInterface (or call it as you want) which probably won't be more than 10 lines of code itself.

So even by taking this into account by design, you still can use your plugins configuration-less without any comprehension or performance impact.

Pierre.

Yes it is. The plugin

Sylvain Lecoy's picture

Yes it is. The plugin configuration is optional and we all agree on that. This configuration system doesn't have to be a dependency for the plugin system, I can't more agree with you, Earl and all the CTools folks. But the drupal configuration (or configuration of plugin) doesn't have to be a part of the plugin configuration system, it then create the circular dependency, it is the infrastructure to manage the lifecycle of plugins. Context, for me, is needed to select the configuration of plugin.

There is plugins which is part of a same configuration (database plugin for instance) but which doesn't need configuration. However, they still need to be loaded in a particular context and not in another (for instance a testing context).

But I think anyway we need all three first. The plugin infrastructure doesn't need configuration or context, it can live without (and i'm happy with that). Then the drupal stack, in a context, load up a configuration of plugins, then inject if needed the context and their plugin configuration.

I have to apologies, after a

pounard's picture

I have to apologies, after a long chat on IRC with neclimdul and Earl, I pretty much agree with them on some problematic here. My diagram upper pretty much helped, I think, for figuring out this. This topic is somewhat basied and questions are not straightforward, IMHO. Maybe a finer granularity in future questions could help.

That said, I'm pretty much sure with a finer design split the circular dependencies would be resolved without event worrying about A, B or C here which are basically built on statements that take the hypothesis the configuration is only a variable storage. For 4 and 5, don't agree, configuration (a finer grain of it at least) can be part of the design without any circular dependencies for the rest, as soon as you isolate the specific configuration implementation that provoques it.

Pierre.

Comments

gdd's picture

I don't have a lot of experience in this field, either in general or drupal-specific, and I find it very difficult to comment on the proposals at hand without that. I'm not really an architect by trade. In general actually I am really architecture-agnostic. I want something that works, performs well, and that the community can get behind. If it has rough edges here or there (like its inconsistent, or its not OO, or we need some one-offs) I'm not really concerned. However, here are some comments I do have

  • I think that while the goal of having one plugin system to rule them all is intriguing and enticing, but I don't think it is necessary, and I think we could do a lot by unifying the stuff that comes after Drupal is already up and all our dependencies exist and punting or doing one-offs on the stuff we're worried about depending on each other. I was thinking about a one-api approach for config as well, but realized that while writing a persistable data system is a well-understood problem, writing one FOR DRUPAL is not, and there are a ton of Drupal-specific cases that can get really tricky (translations, multi-site, performance tweaking, etc.) It's entirely possible that after we work with a divided content/config system for a while we will find places that truly do cross both sides and we can work on bringing them together. This is much safer than trying to shove everything into one hole and coding yourself into a corner.

  • The same goes pretty much for the concept of making all plugins always take the same arguments in the constructor. It's a nice goal and consistency is good, but if only 50% of the use cases actually need both objects, then it seems kind of silly. (I don't know what the number is.)

  • All that said, making things as consistent as possible is good, and while if we can get things down to a few possible plugin types that would still be a HUGE win. Having four plugin systems is still much better than having 12.

  • I don't think config really needs plugins. I had always thought it would go on the bottom, and I had planned an extremely simple dependency injection-type plugin system for it. Something along the lines of

$formatter = new jsonFormatter();
$reader = new localFileReader($formatter);
$writer = new localFileWriter($formatter);
$config = new ($reader, $writer);

With an interface defined for the pluggable pieces. (Note this is the most basic pseudocode possible, have not started writing ANYTHING, this could be totally stupid, etc.) I have already talked to several people about the possibility that very early in the bootstrap we can simplify things by just forcing it to local json or whatever, until you get to a certain point. Regardless, I think config not relying on plugins is perfectly fine.

  • The whole thing of plugins needing config instinctually seems confusing to me too, but its quite possible I don't get something there.

  • I don't know if this will ever get solved in any meaningful way until we dig in and start writing some actual code and see what is happening. Lots of other systems have pluggable architectures, but Drupal is a unique beast in so many ways. I don't think we actually need to make this massive architectural decision right now. Lets start building the context and config systems (which are much better defined at this point) and as we need plugins we start building them. Maybe we do three implementations of each and test them. Which ones perform, what problems get encountered? We will learn a metric ton in the process and it will inform this decision in ways we can't even predict right now. If we end up with four systems at the end and we need to ship, can anyone really tell me this is not an enormous gain?

Hear, hear! I don't know if

Boris Mann's picture

Hear, hear!

I don't know if this will ever get solved in any meaningful way until we dig in and start writing some actual code and see what is happening.

In other words -- everyone with great ideas and passionate for and against views above, it's not going to mean anything until you start writing some code. So git to it!

neclimdul has a sandbox with

merlinofchaos's picture

neclimdul has a sandbox with a rewrite of the plugin code based on an upgraded CTools architecture started.

And has had that code for

EclipseGc's picture

And has had that code for months (In one form or another). He's been tirelessly working on it trying to get it to a good point the handle the situations we've been discussing. In short, we have code.

http://drupal.org/sandbox/neclimdul/1112184

I kind of suspected there was

Boris Mann's picture

I kind of suspected there was some code. So take that message as a pointer to other folks in the discussion to also do some rough prototypes.

+1 The sandbox was definitely

neclimdul's picture

+1

The sandbox was definitely setup for to prototype ideas we where outlining. Its helped us ground those ideas, expand details of some features and interfaces and help communicate them to other people.

I've also been helped by other peoples projects and sandboxes in refining some ideas. Also the points from the plugin system review thread have guided us. So more sandboxes and code is very welcome in helping everyone improve and reach on consensus. Code's like a picture in that it sometimes is worth a thousand words.

Note: I don't know that Crell will be looking to directly pull any of our changes. I've told him I'm comfortable throwing this prototype out and you should be ready to accept the same. We're prototyping not making a blessed implementation.

Yes, and this code is nice,

pounard's picture

Yes, and this code is nice, for the most. Like it!

Pierre.

Leave out the plugin system for now?

Dries's picture

I've started reading this thread; I haven't read all comments yet so at the risk of commenting too soon, I'd like to suggest the following.

The goal of the initiatives is "configuration management" and "web services". While I understand the value of a unified plugin system, and while I appreciate all the work the CTools team has done, I'm not convinced that we absolutely have to pull that into the equation at this point.

Both "teams" have some really good points, but Drupal works without a unified plugin system. It always has, even if that might be a big ugly or cumbersome.

I'd personally leave the plugin system aside for now and focus on the Context/Configuration problems, which are more clearly defined, but also more important. While we do that, we can certainly keep an eye towards how we can make our plugin system more consistent in the future.

Can we try to do that?

(Also, let's remember to be civil to one another; we're all trying to do what is best for Drupal. It is good to see all the passion. Don't forget we're all leaders in this community; as leaders we need to foster the right culture, even when things get heat up.)

I think that this approach

neclimdul's picture

I think that this approach would actually be counter productive. There are a lot of people trying to move forward with plugins right now and I think if there where some leeway to move forward documenting and designing the lower levels of a plugin system we can take advantage of the interest and be ahead of the game when it comes time to integrate with config/context.

There are a lot of bright minds pouring over this right now who may or may not being interested or have domain knowledge in config/context. I know I fall largely fall into that category. Tabling the issue is just going to dissolve a lot of that interest.

Furthermore, I think because we're locked into discussions and being told to hold back, people are butting heads over ideas and problems that sound big but at the end of the day may just boil down to someone providing a 5 line patch.

And the requirements that are at the crux of most of this discussion probably are not actually blockers for the core of the development but just a detail of the public API that we can generally address and refine and provide feedback to config/context as details of their implementation solidify.

Anyways, I'm not volunteering for this or asking for a separate initiative or anything just asking and proposing a path out of this circular discussion before this very passionate and motivated group gets completely burnt out and their input is lost in the final implementation.

I don't think that what's

webchick's picture

I don't think that what's being said here is to not work on plugins. What's being said is to not block the web services and configuration management initiatives on having a robust, unified plugin system in core, which is something we don't currently have and could be months away. It's possible to move forward without this system, and holding up progress of everything else on having it both in place and bullet-proof is incredibly risky.

The idea is to help unblock the circular trinity dependency here, which was the original reason for this thread, not to stop productive discussion/brainstorming, if it's happening.

I totally agree and maybe I

neclimdul's picture

I totally agree and maybe I misunderstood Dries' post. I didn't have any idea that plugins where the roadblock to other things as Crell had previously been quite clear that plugins where a late cycle WSCCI integration.

To make a terrible psuedo gant chart of it, I keep getting the feeling people want because of a dependency on context the only way forward to be:

Config -----|
            |
Context ----|
            V
            Plugins

Where I see it as being:

Config -----|
            |
Context ----|
            |
Plugins ----|
            V
            Integration

If we're all in agreement that plugins don't have a requirement on context for the sake of moving discussions and design forward I totally take back my previous post.

The idea is to help unblock the circular trinity dependency here, which was the original reason for this thread, not to stop productive discussion/brainstorming, if it's happening.

Yeah you're right but it ended up pulling along the baggage of the previous plugin discussions and the wording of the OP sort of pitted the two view points against each other leading to the ensuing debate which is unfortunate because it made getting an answer to the actual question very difficult. Maybe we can try and get that back on track.

Possible solution

neclimdul's picture

So, a core concept of people from the CTools module are trying to bring to the table is the idea of the plugin system being the discovery of plugins. That's been met with a lot of resistance but maybe if I rephrase it we can reach some consensus and find a solution to the problem.

PLEASE DON'T ARGUE CTOOLS HERE. Lets discuss the the circular dependency problem and this proposed solution.

So, lets imagine a theoretical plugin framework that consists of more than one system.

At its lowest level, there is a plugin registry that allows you to get lists of plugins and retrieve meta information that could possibly consist of class names, function names, strings, what ever. We'll call this meta information a plugin definition. Plugin definitions can be registered in a number of ways like hooks. Developers won't interact with much other than dropping a class in a file and returning an array from a hook or something.

Above the registry, is a retrieval layer. This layer maps configuration and context to a plugin, instantiates the object, possibly caches, etc. This layer can query the registry for information if it needs it and will be the interface programmers generally interact with.

Now at this point we can see the retrieval layer obviously has a dependency on configuration and context. That's pretty straight forward but the registry doesn't. Its pretty much completely self contained and just deals with "how do I know about plugins."

So, here's where I see the solution. Configuration can implement its own plugin retrieval layer without a requirement on context or config using the registry layer. With this setup, we can share a lot of logic and conventions of plugins but support both workflows.

I will note that I believe pounard has a mapping layer prototyped that addresses both workflows but I'm hoping this more simplified version of the solution will get some structured conversation started.

At its lowest level, there is

donquixote's picture

At its lowest level, there is a plugin registry that allows you to get lists of plugins and retrieve meta information that could possibly consist of class names, function names, strings, what ever. We'll call this meta information a plugin definition. Plugin definitions can be registered in a number of ways like hooks. Developers won't interact with much other than dropping a class in a file and returning an array from a hook or something.

This registry would still depend on knowing what modules are enabled and which are not. This information might be stored in a storage layer that needs plugins to be loaded.
Somewhere above I made a post about different possibilities to solve this. What do you think?

I glanced around and had been

neclimdul's picture

I glanced around and had been skimming the discussion but don't remember explicitly seeing your solution. I didn't see it as a problem as there's a solutions for this already in my sandbox but it didn't fit with Crell's hooks only model so I left them out of my description to try and avoid CToolisms as much as possible. Basically something like sites/*/plugins with a PSR-0-esque file structure and files containing both definition and implementation(or just definition with implementation being pointed to by definition somehow) would solve this.

There are probably a lot of other solutions so I think we can assume that problem solvable and leave it to a deeper implementation discussion.

Are we on the same page then for the rest of the design then?

The "solutions" are here,

donquixote's picture

The "solutions" are here, points 1a), 1b), 2), 3)
http://groups.drupal.org/node/150149#comment-504784

This is still straying off

neclimdul's picture

This is still straying off the topic I tried to set. The solution I outlined seems to fit 1a but generalized because more systems that config will likely need to exist pre modules.

What about the overall architecture?

If we have this PSR-0-esque

donquixote's picture

If we have this PSR-0-esque sites/*/plugins, then how do we let one project ship with a both module and a "boot plugin" included, to be easily downloaded with drush dl or ftp upload and unzip?

Also, I dislike the name "plugins" for this. For me this name suggests that anything that is a plugin should live in this folder. As far as I understand, there would be other plugins that do not live there, but in modules. Depending what we put in there, I would prefer either sites//boot, or sites//libraries. 'boot', if we use it for anything that is to be available before we know about modules. 'libraries', if we use it for anything that is not a Drupal module (this would include things that already are libraries nowadays).
But anyway, I'd rather see this in the modules folder than anywhere else, so it can be shipped with module projects.

Of course, if we let anything

donquixote's picture

Of course, if we let anything in settings.php refer to a specific module, then it has to specify the path. This might conflict with the path we already store about this module, if the same module is available at two different filesystem locations. As pointed out in 1b), we have no choice but to always give priority to settings.php, and let it override whatever we have in the system table (that is to be moved into the "configuration" layer)

Adding more "you module is

neclimdul's picture

Adding more "you module is running but modules aren't available" run levels is a recipe for ruin. hook_(boot|init) are painful, and brittle and we'll just be extending this concept into even scarier waters.

Also, I have doubts about

donquixote's picture

Also, I have doubts about introducing some PSR-0, if at the same time the modules system is not PSR-0 at all.

PSR-0 is terrible for us. I

neclimdul's picture

PSR-0 is terrible for us. I probably shouldn't have even mentioned it but sort of aligns to the plugins/module/type/foo.inc or plugins/type/foo.inc sort of structure that would work well for us.

Ok, so we have one registry /

donquixote's picture

Ok, so we have one registry / discovery layer, and one retrieval layer.
In the registry layer, modules or other things can define an instruction for how to create a plugin. This could be the name of a factory function, with some parameters. Some of the parameters could be wildcards, that have to be filled with context or configuration, or with user-specified parameters. This means, the registered plugin definition tells us if this specific type of plugin depends on configuration and context or not.
The registered definition could also contain a form array for manual configuration of plugin instances.

Now for instance a module could check what plugins are available for input filters, and provide a form to assemble and configure input filters to an input format, using the forms provided in the plugin definitions. The result is stored in the database, tables that were defined by this specific module.

Now when the input format is to be used for text formatting, it will look into the database to find the plugin settings and ask the retrieval layer to build the plugin objects. The retrieval layer will look into the registry to find the factory function (which can by default be a simple create_class_with_name) and the definition array. The retrieval layer will merge the settings and the context or configuration or whatever stuff it has available into the definition array, and invoke the factory function.

Dependencies?
- The usual discovery / registry layer depends on knowing the module enabled status, but we could provide an alternative implementation that does not. It should not depend on context (as in "request and session"), but some modules might provide different plugin definitions depending on things in the database or configuration. For instance, a module that allows the admin to manually create input filters, has to be able to look into the database for plugin discovery.
- The retrieval layer does need to provide context and configuration for some plugins, but not for others. We could even have more than one PluginRetriever object, each parameterized with a different context.


There are some things we might call "plugins", where the discovery layer is not needed, and the retrieval layer can be a simple hook. The retrieval layer will just module_invoke_all() the hook, with some parameters, and it will return the plugin objects. No manual configuration, except maybe some per-module configuration.

So yes? Can I ask that we

neclimdul's picture

So yes?

Can I ask that we please stop focusing on implementation details and discuss the design? How these concepts happen is not as important as if we can agree to the responsibilities and requirements I set out in the design and them as a basis for moving forward. If we can agree to a solutions to the the trinity dependency problem we can move past this nightmarishly long thread and start discussion those individual implementation details separately.

Ok. I am copy/pasting my

Sylvain Lecoy's picture

Ok. I am copy/pasting my previous post here mainly because of the width limitation when comment depth being too big.

Can you describe here how you see the bootstrapping ? Which plugins precisely are loaded, and when. Then when, for you, the configuration needs to be loaded and what it contains exactly ? Some diagram or use case will certainly help to get your point.

Only local images are allowed.

Configuration is concerned with the management of all software products (modules, plugins) throughout a request. This includes the Session plugin, rendering plugin, persistent plugin and caching storage plugin to load (in this minimalistic example).

A particular configuration can be in the case of a testing context, which you may not need a real database plugin but a stub plugin. Versions of these plugins are a same implementation of a type. They can be interchangeable together.

Thus, configuration deals with both the product space and the version space. Software objects and their relationships constitute the product space, their version (or variants) are organized in the version space. A configuration combines product space and version space. In particular, it has to support the consistent selection of versions across multiple configurations.

Only local images are allowed.

From a Request (1) will be deducted a Context. The configuration API will then be able to retrieve the configuration needed and do the correct mapping (2). This operation will ensure that the plugin will be loaded for this particular context and not another (A JSON rendering plugin for an API call, a Stub database plugin for a test scenario, etc.). Note: The step before (3) is a short hand for every plugin loaded individually, I assume that plugin lazy load because of the autoloading capabilities of PHP and not in a single step as it suggest in the figure above.

The plugin will then retrieve (if needed) the configuration for each lazy loaded plugin by calling the configuration API. A configuration object may contains a configuration dictionary and allow properties to be updated via this object.

If we compare with the classical example of the Abstract Factory Pattern:

Only local images are allowed.
See in lage: http://drupal.org/files/issues/ConfigurationAPI_1.jpg

Intent Provide an interface for creating families of related or dependent objects without specifying their concrete classes.

I make the assumption that we want:

  • That the system should be independent of how its products are created, composed, and represented.
  • That the system should be configured with one of multiple families of products
  • That a family of related product objects is designed to be used together, and you need to enforce this constraint.
  • That you want to provide a class library of products, and you want to reveal just their interfaces, not their implementations.

Participants

  • AbstractPluginFactory: declares an interface for operations that create abstract product objects.
  • ConcreteConfiguration: implements the operations to create concrete product objects.
  • AbstractProduct: declares an interface for a type of product object. Storage plugin, rendering plugin, etc.
  • ConcreteProduct: defines a product object to be created by the corresponding concrete configuration factory. Implements the AbstractProduct interface.
  • Client: uses only interfaces declared by AbstractPluginFactory (PluginAPI) and AbstractProduct (type of plugins).

Of course the model is an example for four hard-coded plugins type. We want it more generic so we can safely remove the abstract methods CreateProductCaching, CreateProductRendering and so on. So, the configuration instanciate the concrete plugin factory thanks to the context:

Context #A
context->http_accept = application/json
context->http_session = 0939FU1S83CORcx
context->http_if_match = "ETag: 01930840"

You then give the configuration API the context object. The configuration API lookup for a hash of the context to see if there is a cached configuration in persistent layer for this context. KEY = context->getHash(). If not the context object get injected into plugins factories to get the correct implementation on-the-fly. The Rendering Abstract Factory will return a JSON plugin cause the factory knows to look for the http_accept key. The Session Abstract Factory will return the correct implementation of the session plugin, and so forth. The configuration API then save this configuration with the hash as key in the persistent layer, for further requests.

And here precisely we have to make the choice of not having swappable configuration for the bootstrapping configuration. The most performant key/value system is not a serialized JSON file but really a database system. Moreover, its important to treat this as a management issue, that's why its not needed to be "SCM friendly", developers should never care about this at development time.

The downside is it can be better to put it in APC or Memcache, and this is where I agree that having plugin as a base is better. But we can find a work-around, the sys-admin can specify in the settings.php which plugin to use for the configuration system, and I believe this fix the problem for pluggable configuration.

Only local images are allowed.
See in large: http://drupal.org/files/issues/PluginFactory.jpg

While you now see that there is no need to implements the AbstractFactory as an interface (in the PHP sense), we can have a single PluginFactory, which instanciate an AbstractPlugin based on a configuration, which is selected from the context. Here is an Abstract Model :
Only local images are allowed.

And here the concrete model showing up the dependencies.
Only local images are allowed.

We can see the APCConfiguration, or DatabaseConfiguration as a concrete plugin of a type "Configuration". And that's here mainly people disagree, because effectively, in this case, the plugin API has to be before. But I don't think this is necessary, the concrete plugin can be written in the settings.php by a sys-admin. Swapping between concrete Configuration will not be something used everyday.

That's why in my opinion, I think the dependency order is Context -> Configuration -> Plugin

Any feedback ?

Sources of ECore Diagrams can be found here: http://drupal.org/sandbox/SylvainLecoy/1170126

A "Reasonable" Compromise

tinyrobot's picture

Sylvain thank you so much for these diagrams, they really put into perspective a lot of your comments.

So let me summarize things trying to use WSCCI definitions so I can make sure that we all are in the same page.

1) As Sylvain’s diagram shows, the first thing would be to load the Context API and create a Context. This Context is being initialized with the Request Context, and that is it.

2) At this point we load the Configuration API. The Configuration API loads a Map.

3) Then we load the Plugin API.

4) Somehow the map gets interpreted, the Plugin API loads the necessary plugins and things get executed as they are supposed too (I am being vague on purpose).

5) After all the plugins execute; we should have what we want to return.

In his diagram, Sylvain shows that plugins can be complex by using the Configuration API to load other plugins given their own maps (plugins can augment the configuration map).

One thing that I want to point out is that Configuration here is synonymous with a map of plugins (is that correct, Sylvain?) The other interpretation of Configuration is what Sylvain at one point called Settings. Storing and reading settings is what hejrocker is working at here (http://groups.drupal.org/build-systems-change-management/cmi), and it will simply be a plugin.

So, as we can see, all of the relevant APIs in this discussion (Context, Configuration, and Plugin) are loaded very early before anything “major” happens, but here is the argument that many people are bringing as I understand it.

- So what if the Configuration API, or Context API wants to use plugins?

So lets load the 3 APIs at the beginning, before any of them are used. This does not solve the conundrum, but it makes them available if we want to use plugins as part of the Context or Configuration APIs.

I agree with Sylvain, that Context is important to choose the correct map (Configuration), but by Context (according to his diagram) he means Request Context, and the Request Context is much simpler than the whole Context infrastructure as described in the WSCCI definitions

So here is what I propose:

1) Load all 3 APIs (Plugin, Context, and Configuration)
2) Build the Request Context
3) Give the Request Context to the Configuration API so it can get the correct Map. (Configuration API can use plugins, and it can even use itself if it wants a complex ordering of plugins in its internal implementation)
4) The more complex Context gets initialized (it can use plugins, and Configuration maps if it would like)
5) the rest happens as in Sylvain's diagrams

This is the best compromise that I can come up with after reading the majority of this thread. The only “one-off” thing that comes into play would be when building the Request Context.

I hope this all makes sense.

x

Yes it makes sense, thank you

Sylvain Lecoy's picture

Yes it makes sense, thank you as well for using the correct definitions and putting words on my diagrams.

I think you highlighted an interesting concept, as pounard did as well: loading the 3 API at the beginning. Making them 3 working independently and then wire them up.

That's why in my opinion, I

pounard's picture

That's why in my opinion, I think the dependency order is Context -> Configuration -> Plugin

+1 that's the order I proposed in my model (except I removed the context, but it should be before configuration, or at the same time at least).

Pierre.

Just want to say, I'm ok with

neclimdul's picture

Just want to say, I'm ok with this.

Because it fits here and not because I think pounard doesn't understand my stance, I want to be clear on why I and others have been very vocal about not requiring context or configuration as dependencies. I was trying to explain this in my other post but we want to limit the parts of our plugin system that do require other systems. There absolutely will be a layer that does require our context/configuration API's but most of the core should not and if we do things right should allow us to build plugin systems that do not require it. The requirement is arbitrary and a Drupalism imposed to keep core consistent. Thats fine. But really it is unrealistic to expect of every contrib and custom site implementation to work like this and will for no real gain limit developers ability to use the core plugin code.

For the sake of moving things forward, lets accept this dependency order and start investigating what that will mean for WSCCI, Configuration, and other initiatives.

I totally see why you'd want

pounard's picture

I totally see why you'd want to keep it dependency free. In fact, it's a good thing to decouple components. But at some point, when building a full API you may need to tie some of them together in order to make things simple.

But don't get me wrong, I really understand. My point was all about having a really light configuration API (that is not about variables at all). Putting variables as a dependency to anything would be a dawn wrong Drupalism, building a configurable plugin system based on a really lightweight independent configuration layer is not a Drupalism, it's just putting the configuration inside the design and easier its usage (which is probably the real debate we should have). Speaking about context, it's pretty much more business logic oriented and should probably be put aside this "configurable or not plugins" debate.

The reason why I'd debate this much is because if "configuration" in D8 becomes only a synonym to "variables" then no problem at all will be solved (why use a configuration object instead of actual variable_get()?, this wouldn't bring any good). If the build system & change management comes to build only a new API for variable_get() and variable_set() I'd probably say they have done a lot of working for nothing better.

But totally agree about the fact it needs to move forward, I'm just stopping right here, for whatever reasons I was arguing so much was probably not worth it in here, it'll probably come back later in another context.

My opinion is that in here is being discussed a whole complex problem and that's quite an error. That should be divided, as in "divide and conquer" before even starting worrying about dependencies. Make a configuration layer that works alone, make a context layer that works alone, make a plugin system that works alone, they try to tie it together after, and everything would probably be so much easier.

Pierre.

+1

neclimdul's picture

+1

Context -> Configuration I

donquixote's picture

Context -> Configuration

I disagree.
Let's first agree on what we understand by configuration.

  1. A persistent storage layer, that stores things like: Which modules are enabled, and some things that would otherwise be stored in the variables table. Maybe also roles and permissions. This stuff should be the same for every request.
  2. Various subsets of the data stored in configuration. These can be specific to a request, to a user, to a role, etc, or they can be dynamically generated from different parameters. These things do not override the basic configuration from (1). For instance, there is no user- or page-specific module enabled status. Or, there are obviously user-specific permissions, but these do not conflict with the global permissions table from the configuration storage layer.

Now, we could also distinguish different pieces of context:

  1. Stuff that we need to decide which multisite to use. I agree, this comes before anything else. But I think we should be careful to keep this small and simple and not dynamic.
  2. Raw request and session data, as we get it from PHP. Imo we should not look at this to load the configuration layer. No single value in the persistent configuration should depend on this request data.
  3. Interpreted request data: Instead of a raw url, we now have a language + drupal path (de-aliased) and a menu router item. Instead of a raw session, we have a user. Etc. Again, we should not let the persistent configuration layer depend on this. But of course we can look into this configuration layer, to find settings specific to the user's roles etc. For instance, the theme that is to be used for this page. But all these user- or page-specific settings should be stored either in the "context" object, or in some other object accompanying the request.

Btw, I still don't understand which of the above we want to name "context". It's not a very meaningful name, imo.

Let's first agree on what we

neclimdul's picture

Let's first agree on what we understand by configuration.

Done. What ever we decide it is. If we agree about much more then that in this thread we will be talking for months and will undermined an entire core initiative.

Now, we could also distinguish different pieces of context:
...
Btw, I still don't understand which of the above we want to name "context". It's not a very meaningful name, imo.

Um... please understand the core concepts of WSCCI or this discussion is going to be very hard. I think the goals and constraints of Context are clearly stated in the WSCCI Definitions and I think that's pretty well accepted. The details will be ironed out in issues and future threads as development kicks off.

From the WSCCI

donquixote's picture

From the WSCCI definitions:

The context of a request is all of the data and metadata that derives from the incoming request, either directly or indirectly. It represents the environment in which code is being run.

I have a problem with this statement.
It does describe quite well what happens in current Drupal: Something in hook_init() or during theme negotiation knows about the current request, and sets a global (or static) state variable. These variables then represent the "environment in which code is being run".
That's the way it works today, and maybe it is just not realistic to change it in Drupal 8.
But, I really would prefer something where the request data does not affect "the environment in which code is being run".
Of course this depends on what we understand by "environment". In my understanding, anything that is global is "environment", whereas anything that is passed around as parameters is not "environment".
Imo, anything that does depend on the request should ultimately be passed around as parameters, instead of infesting global state, so finally we could make it possible that one Drupal process can handle more than one request (there was another comment about this, also by me).

I strongly assume we will not get to this point Drupal 8, and maybe not in D9 either. The current theme, the current language, the current user, will probably all be in global state, even if they are made available via the butler object. But I think one should at least try to work towards this idea, whenever we make big architectural choices, and not move further away from it.

Maybe this should be a new thread then..

Configuration is the problem

tinyrobot's picture

I agree that Context has been well defined, but Configuration has taken two meanings in this thread. The Configuration Management initiative is a rework of the variables system, but what pounard and Sylvain (and maybe others) are calling configuration is a mapping layer. I think I understand what Sylvain means by that, and I don't know if pounard means the same thing, but I guess that this should be a separate initiative if most people think that it is relevant.

x

I've talked to pounard and

neclimdul's picture

I've talked to pounard and looked at his code and don't think that's what he's saying. A plugin mapping layer is what he's talking about but configuration is a separate thing injected into it and used as part of that layers logic. What ever configuration ends up being, we can inject that and adapt the mapping layer to work. I think his diagram may have even allowed for different configuration schemes.

The point is, integration of Context, Configuration with a Plugin system /will/ happen otherwise the benefit of the 2 initiatives will be drastically limited. So not really up for debate.

Also, I want to reiterate the point of /this/ thread was to sort out how to resolve the possible circular dependency and of Plugins, Config and Context. It seems we've sort of decided we want to keep them as loosely coupled as possible and move towards some sort of "Context -> Config -> Plugin" like dependency. That's a good goal at this point and implementation may lead us elsewhere but I think we /all/ agree we need to move forward.

Basically a "configuration"

pounard's picture

Basically a "configuration" is just a set of key/value pairs, hierarchical or not. I'm not sure Sylvain does think exactly the same thing, but I think he sees the dependencies in the way I do. May be he decoupled a bit more with contextes I suppose.

If we think about it, a context is no much more than a set of key/value pairs, with a bit of business logic in it. That's why I'd see a basic context layer as an extension of a basic mapping layer. I would do this only in order to share code. I'd put the simplest of the two being the dependency to ensure minimal requirements (purely technical point of view). While configuration is not business tied and while contextes may be, it seems to be the right way.

Plugins are nothing more than concentration of business logic functions that may need or may not need be configured, and that may or may not need to use contextes. While saying this, I'm saying than a basic plugin layer does not need to depend from any of the context or configuration, but plugins are business functions, and configuration and contextes are basically no more than value objects that carries some business information (but in a generic manner, so not dependent from the business layer itself - a really low level tool). So if I'd do it, I'd put the plugin layer being dependent on the two other: but only in order to provide access shortcuts for developers.

Another good stuff is that if it's design centric, then you can safely make configuration and context being transported from place to place, plugin to plugin, page to page, anything to anything in an elegant and automated manner right from the start. Doesn't mean it has to be this way neither.

This doesn't mean than plugin really depend on context on configuration, that just means than it could exist a tool hierarchy to enforce people to use the one provided, in order to ensure that any other tool around (configuration export, context hash computing for caching handling and stuff) would be used correctly if needed.

In the end, the three can almost be non coupled at all. But I'd say that business logic needs context and configuration pretty much always, or let's say really often.

But yes, please move forward :)

EDIT: That said, I'm not participating to this issue anymore, because as Crell said "At this point I think most people want you to just leave the group entirely" - until someone proves him wrong at least.

Pierre.

I am probably not the best

Sylvain Lecoy's picture

I am probably not the best placed to say that but if I agree that a configuration store which modules are enabled, this is not the same thing for plugins. You can have different plugins which implements the same type of plugin.

For a caching plugin: APC, MEMCACHE, DATABASE...
For a rendering plugin: JSON, AMF, HTML4, HTML5, SOAP...

While I submitted the idea of not having all modules loaded here (for example Admin modules, or UI helpers modules) for a request, people nicely explained to me that it would be too complex and I understand, but for plugins we haven't the choice, we need to load the correct one for a particular context (or request).

Mid term planning

eigentor's picture

Having just read Larrys post and the first few comments:
How about planning more mid-term? The general goal sounds absolutely marvellous. But we also have the goal to realease D8 in not three years, but rather in two.

So maybe you can figure out a way how to build a part of the system that is more of a common-sense approach to enable to really get nuts in D9 and implement all the bells and whistles.

Still: I guess reality will do that for you and cut the wings a bit. The thing sounds like rewriting large parts of core, and we all know this has serious time of sanitizing after it.

Life is a journey, not a destination

In conclusion

Crell's picture

I think this thread has run its course. After discussing with Greg Dunlap, Angie Byron, and folks from the ctools team, we've come to the conclusion that, basically, we're not going to resolve this issue right now. :-) Instead, the game plan will be:

  • Get the new configuration system in place first. Any swappable functionality there will be implemented one-off. (Greg is lead on this part.)
  • Get a context system in core, along the lines that are being worked on in the Butler module and originally described in this thread. The logic to determine which context handler responds to which context will be implemented as a one-off. It will rely on the configuration system for storing its mapping data.

Those two tasks will be the focus for June.

Once those two are in place, the CMI team will continue to develop the configuration system while the WSCCI team will work on plugins. Plugins will initially be implemented on their own, not intended to sit "under" either Config or Context. Whether or not all plugins will take Config and Context objects is an implementation detail we'll figure out later.

Once we have all three systems in core, we will revisit the question of whether or not the one-off logic in the configuration and context layers would be better off replaced with a "reduced mode" of plugins. I have a feeling that once we have working systems in core we'll all have a much different perspective on how feasible that is and whether or not it's a good idea.

So let's for the moment shift from long-view to shorter view, and start cracking on the context problem. :-) See the project linked above where there's a few open issues. I'll try to provide a more useful description on the project page shortly, but that's where we should be focusing our energy for now. There's a few specific open issues that we should address, rather than trying to revisit the basic architecture. That part we're already reasonably set on, I think.

Simple Object Oriented Solution

Sheldon_Parnes's picture

I just started using Droople and have not done much with out but I've solved many similar issues with the use of Singleton Classes. I am not worried about whether you want to use factories, abstract factories, or just an array of key word value tags.

In general you would start of with a Singleton context manager, plugin manager and configuration manager. First any context information that is owned by the core and is static or sacrosanct can be be accessed through specific function calls through the context manager, this will allow us to define the context managers, configuration manager and even the plugin managers core functionality in terms of plugins. Each singleton would allow a plugin to be registered with the class to control all other queries, which is just passed directly to the plugin. No plugin is generated until a query is generated. We could of course replace the default context manager, plugin manager or configuration manager plugin at anytime but it would probably be better not to change horses in midstream. The core default plugins could of course be

Note that you will need to defend what goes into these Singleton classes with your life because it is very easy to inadvertently break the encapsulation needed to support plugins.

I won't go in to any more detail because I don't know if this will ever be read.

Governing autonomous elements

AndrzejG's picture

All initiative discussions seem to be halted without clear results. Maybe some new questions are necessary?

For me, it is not clear what are the centralised system for, what they have to do, and why.

For example above thread is about centralised plugin system but focused on the particular choice. I would ask what has the system to do to enable plugins be autonomous? Or what is the minimum unification or centralisation absolutely necessary for governing autonomous plugins? What is the minimum function of the system with respect to the governed plugins? And why?

The same, mutatis mutandis, concerns centralised context system. What minimum centralised function is necessary to enable blocks, plugins etc. to autonomously respond to various contexts or act independenlty of any context? Maybe the idea of broker should be more clarified.

Some updates please

mandela101's picture

I have been following this thread from the beginning and it seems like there were no immediate solution to the problem.

I would like to hear some update from the team on how the project is going along.

juicer

Status update

Crell's picture

http://groups.drupal.org/node/165059

This thread of doom went on long enough. Let's let sleeping threads rest. :-)