Butler in Chicago

We encourage users to post events happening in the community to the community events group on https://www.drupal.org.
Crell's picture

I'm a bit late in getting this out, but such is life when moving and attending a wedding. :-)

We had a good Core Conversation on Butler in Chicago. Sadly Dries ended up not being able to attend, but just about everyone else was on board with the roadmap. The roadmap itself hasn't changed much since Copenhagen, but things are moving again. There was also considerable talk of very, very interesting tie-ins to the configuration/deployment core initiatives, as having a more robust pre-database configuration option should help resolve a lot of the challenges we were looking at.

Slides from the presentation are attached below.

Next steps:

Context

There is a partially completed context system in the Butler project on Drupal.org. Right now the handler registration system and response system are in place, but there are still some gaps and bugs. I will fill out some issues in the issue queue shortly. If you want to hep with the Butler project, this is an excellent place to jump in. Just remember to read through the architecture discussions in this group first. :-)

On a partial tangent, David Naughton (nihiliad) and I have been talking about leveraging the PECL HTTP module for interfacing with the HTTP request. That of course introduces a dependency on PECL... unless we have a user-space implementation of it. He's working on that now as a stand-alone project.

The idea is that we could include a user-space implementation of a known and tested PHP module that other developers are already familiar with in Drupal core, and use that. But, because it's OO, we can lazy-load it. If you install the PECL module, though, suddenly that part of your code becomes C, not PHP, which is faster. It probably won't have a huge impact here, but it's a trial balloon to see how viable this approach is for other PECL modules. Stay tuned. :-)

Plugins

OMG everyone seems to want this, even moreso than the unified context system. I still believe that we need the context system first and then we build plugins on top of them.

However, one of the things we discussed is not yet-again reinventing the wheel. In particular, while I have code partially working and we have the ctools model available to us, it is worth our time to try and investigate other systems to see how well and how cleanly they address the problems we need to solve.

Fortunately, a number of people stepped forward to help with that research. Specifically:

  • Sam Boyer and Dmitri Gaskin are researching Symfony 2.
  • Marcus Deglos (Manarth) is researching Kohana.
  • Justin Randall (bejeebus) is researching Zend Framework.

And of course we should be looking at ctools as well. Mind you, I don't foresee us directly using any of the above systems per se. We're not introducing a dependency on Kohana. :-) However, we do want to try and learn from other systems so that we can build a more robust, thought-out system ourselves. Researchers, please post your results in this thread.

So the question is, what exactly do we want to research? I see our plugin system needing the following from a 10,000 foot perspective:

  1. Definition of a type of plugin (eg, Cache)
  2. Definition of a plugin that supports that type (eg, Memcache)
  3. A way to define and store configuration for a particular instance of a plugin (eg, Memcache + the bin it is attached to).
  4. A way to retrieve a particular plugin/configuration pair by name.
  5. A way to define logic for "find me the right plugin for X circumstances". (eg, "cache bin for pages". I've been calling this routing.)
  6. A way to aggregate plugins together sanely (eg, a View and a Feed are simply collections of defined plugins and configuration. Ideally, text formats would simply become a collection of filter plugins.)
  7. All plugins that rely only on context information should, ideally, be auto-cacheable with no further work by the developer. (Blocks will definitely want this, but it also becomes a great built-in cache for Views queries, text filters, etc.)
  8. We need a standard way to expose a form to manage configuration, but configuration should be manageable through a script, too, not just a form. (I like the ctools/views plugin model here, except that the configuration object should be a separate object from the plugin itself.)

Items #1 and #2 are such perfect fits for a PHP Interface and a PHP Class (respectfully) that it's not worth discussing other alternatives. The rest are still somewhat open.

To researchers: What we want to know about each system is how it addresses each of the above needs (assuming it does), what's good about it, what's bad about it, and what we could learn from it. For Kohana in particular we should also look into its nested block rendering system, as it seems very close to what we've been discussing.

In practice I suspect we'll still use hooks for much of the definition work. However, one of the high points of the conference was an impromptu meeting with the configuration team, led by David Strauss and Howard Tyson, and Greg Dunlap of deployment fame. One of the things we've been struggling with for Butler is how to store and manage all of this configuration we will need. Most of it requires, given Drupal's current design, either a full database load with lots of queries, lots of really big nested arrays that get cached, or hook invocations that therefore require a full bootstrap. We're trying to avoid all three of those as much as possible.

However, the configuration initiative looks to be moving toward on-disk human-editable JSON files as the primary configuration storage system. That could be loaded trivially simply very very early in the Drupal request lifecycle, and give us read-access to a much richer set of configuration options. If we use that as our primary configuration storage for Context and Plugins, then we can neatly eliminate the most performance-hurting part of the system. It may not help for Drupal 7 backports but it still looks like an extremely promising solution to that problem for Drupal 8.

Some nice supporting links that have come my way in recent weeks for those researching plugin alternatives:

http://components.symfony-project.org/dependency-injection/documentation

http://framework.zend.com/manual/en/zend.controller.actionhelpers.html
http://www.zfsnippets.com/snippets/view/id/48

http://techportal.ibuildings.com/2010/02/22/scaling-web-applications-wit...

AttachmentSize
Butler-phase-1.pdf948.03 KB

Comments

Very exciting stuff. I am

te-brian's picture

Very exciting stuff. I am trying to make time to dive deeper into frameworks like Symfony2 and Kohana (and even Rails) to broaden my horizons a bit. It's easy to get Drupal near-sightedness when your day job is ONLY in Drupal land.

As an interesting thought experiment, there is a lot of talk of separating "Drupal the framework" from "Drupal the CMS". What would be interesting is following that line of thought all the way "down the rabbit hole" to an alternate reality where Drupal is built on top of symfony (or kohana, or whatever) and we build entities, users, fields, our admin workflow, etc. as plugins (or bundles in symfony's case) on that framework.

Obviously not a reality (DX nightmare?) but an interesting thought none-the-less.

I look forward to being able to contribute more as my knowledge base grows and I figure out a way to harvest time :)

1 and #2 are such perfect

sdboyer's picture

1 and #2 are such perfect fits for a PHP Interface and a PHP Class that it's not worth discussing other alternatives.

Except that ctools doesn't require class-based plugins. In fact, class-based plugins are a more recent addition; historically, all the original plugins were procedural. And that's an important reliaty, because we've got dozens, maybe even upwards of hundreds of procedural plugins based on the ctools model.

Looking forward, this isn't super-important; I'm very nearly to the point where I'll concede that plugins ought to just == OO. Hell, the last ten plugin types I've created (all under the auspices of the git migration) have been OO. And really, there are so many additional costs, annoying confusions, etc. associated with procedural-based plugins; even from the start, ctools plugins have been weak userspace OO (they've always had a declaration array containing function names and static values - aka, methods & properties). There are historical reasons that made OK at the time, but...yeah.

All that said, though - we can't just leave all those old implementations behind. We'll at least need some sane docs to help support the upgrade process.

Agreed on documentation

Crell's picture

Yes, we definitely need a clear upgrade path for people used to ctools plugins, both versions.

However, in true Drupal fashion the existence of legacy procedural plugins now should not stop us from building the best system going forward, which I believe means OO-only.

While I believe that many of

merlinofchaos's picture

While I believe that many of the plugins in CTools would be better if they were OO, plugins that are primarily output may not really benefit that much.

My canonical examples are Panels' layout plugins, which look a lot like a theme .info file: They are the metadata which includes a list of regions, 1 or more templates and 1 or more CSS files, and possibly a preprocess.

They can be more, but they almost never need an actual function callback at all.

Advanced Forum goes a step further; it has forum styles that are a package of templates and an .info file that describes some settings, and nothing more.

In both cases, the OO needed to write a class, in my opinion, will detract from the target audience of the plugins: that is, users whose primary goal is to write templates and CSS -- these people should not be exposed to classes.

By the way, in my ideal

merlinofchaos's picture

By the way, in my ideal world, the entire theme system is, in fact, part of our plugin system. Because it has a lot of the same needs and requirements of a plugin system.

To a point, yes

Crell's picture

I agree that much of the theme system could benefit from leveraging plugins, but I probably would not go as far as making every theme function call into a plugin. The overhead for that is probably even worse than what we have now.

As far as procedural plugins, as we've discussed in the past I agree that "template plugins" don't really benefit from having an OO core. However, it's simple enough to emulate the procedural part of them by just specifying the name of a callback function, or template, or whatnot and using a default plugin class for all plugins. I did that once in one of the various iterations that handlers/plugins/whatevers have gone through. That keeps the core system itself simpler and cleaner, and if done properly there's little if any additional overhead for the user/implementer.

As a general rule, if you can easily subsume one use case into another then it is, usually, a better approach to just implement the more robust option and not use all of its features in the simple cases. See also, "N is the only number" (for those who saw my DrupalCon Chicago presentation).

I agree that much of the

merlinofchaos's picture

I agree that much of the theme system could benefit from leveraging plugins, but I probably would not go as far as making every theme function call into a plugin. The overhead for that is probably even worse than what we have now.

This is not what I meant.

What I meant was that a theme is a .info file that defines a package of templates, CSS, javascript and other related files. As such, the plugin system could leverage theme discovery, etc. The idea being that a theme is a plugin; it still can have various theme callbacks (thus emulating a module) and so on.

By the same token, a module is the same thing. It's a plugin that specifies behavior (i.e, lots and lots of hooks). The code to discover themes, modules, profiles, image derivatives is all more or less the same, there's just the matter of ownership and placement of code and different ways of doing it. It might be overzealous to try and turn a module into a plugin, but then again, with just a tiny bit of magic, CTools plugins.inc could, in fact, handle modules. And that magic would be: Not needing hooks to do discovery, but instead an internal list of possible places for the modules. Other than that... .info file parsing and some UI and voila. Well, there's the system table, that'd be handled differently in the plugin case, but that's not such a bad thing.

Maybe I'm being idealistic here, but that's one of the reasons I want to keep the plugin system flexible. I like OO. I think the vast, vast majority of plugins should be OO, but I'm still very leery of making the leap to "must be OO".

What I meant was that a theme

Crell's picture

What I meant was that a theme is a .info file that defines a package of templates, CSS, javascript and other related files. As such, the plugin system could leverage theme discovery, etc. The idea being that a theme is a plugin; it still can have various theme callbacks (thus emulating a module) and so on.

Ah, OK, just making sure. I agree that the theme itself, and probably theme engines if those stick around, are potential candidates for plugins. We'd have to see how far we could get with them but it's certainly worth exploring.

I don't think I'd go as far as module == plugin. That strikes me as moving from "hooks are our hammer so everything is a nail" to "plugins are our screwdriver, so everything must be a screw". I don't think any one system is going to be able to handle every use case we have, but we should aim for a "big three" approaches or something like that.

Mind you, there are ideas bouncing around my head that I've been toying with to refactor the way hooks work, but that's unrelated to butler, context, or plugins. ;-) I'd rather leave modules as "bundles of vaguely related code that integrate into the system", where "integrate into the system" has 2-3 good mechanisms rather than just the one we have now.

Well, the basic structure of

merlinofchaos's picture

Well, the basic structure of theme and module discovery is the same. In theory, if you're using this primarily for discovery (this tells us where our themes are and what they contain) then that very same system should be modules. So, right now I have it as an interesting thought experiment, and not something I would necessarily seriously consider. That said, my goal is that the system should be flexible enough to consider the possibility. Particularly for theme and theme engine and library, which really are plugins in the way we think of them.

Hopping in at this point,

EclipseGc's picture

Hopping in at this point, another really great example of things that COULD be pluginized are administrations. My context_admin module has been doing this for a while, and I'm attempting to illustrate that to some of your team members via reworking workbench some, but the point is that non-generic administrative tools are bad, generic (plugable) are good, and since a ctools style plugin removes the code from the memory footprint in its entirety, we could get back a LOT of what we've lost in the 7.x cycle performance wise. (i.e. imagine removing the entirety of drupal's administration from the memory footprint it runs in)

Also, this reply is out of sync since the plugin ui conversation is happening elsewhere, but in-so-far as what we do concerning "plugin ui" it's worth mentioning that many (most) of ctools core plugins at this point actually accept and use "wizards" for their user-facing forms, so that we can build as robust of a UI as necessary for the end user. Keep that in mind as we move forward with this plugin stuff as well. Relegating us to the views mechanism is unrealistic as the views mechanism is really representative of a particular PLUGIN species' needs, and not plugins as a genus. Some plugins have different needs for "edit" than they do for "add", some plugins within the same species use wizarded forms while others use a simple 2 or 3 field form. The diversity of individual plugins within a given plugin species is quite profound in some instances.

Also, working with merlin's thought experiment for a moment, the wonderful thing about ctools style plugins is that they are essentially self documenting for the initiated. A plugin array definition specifies any plugin specific "hook" (though I guess these are more exactly "handlers" since there's not a stack of them firing) and that same logic can be extended to drupal hooks. Context Admin implements hook_form_alter and hook_node_insert at the moment for any of its plugins that want to take advantage of those core drupal hooks. What's cool about that is that while core calls ALL instances of hook_form_alter() context_admin finds the specific ones that matter for the page we're viewing and only executes them. Again this code isn't necessarily practical on the scale of core, but what is interesting here is that ctools plugin array definitions are probably a pretty good starting point for a potential hook cache. Just saying :-D

I just read the full plugins

pounard's picture

I just read the full plugins handling code in CTools, and there is probably good stuff in it but within a rapid overlook I see an over-engeneering abstract code that takes place itself between the hook system and pure OOP code based on interface using some registries. I might be wrong, but I think it's a good example of "why would we do something simple when we can do it in a really complex and obfuscated way?".

Sorry if this hurts, this remains hell of code (I mean, it works) but I don't see the advantages compared to usage of both hooks and good OOP programing.

Pierre.

I really appreciate you

neclimdul's picture

I really appreciate you taking the time to do that. I'm sorry you did so before I had a chance to finish up polishing a version of the code in a sandbox. I'm not at all hurt though as we're well aware the code is riddled with special cases for backwards support of old features, weird function names that have never been fully cleaned up, old hacks around core and some code that's not even really related in the context of plugins in core.

Also, because it has grown to a state of maturity supporting plugin systems, it has a some features that might not be strictly necesary for the simple use cases. But they are features that where needed for real applications.

Because of all this, I can see why you might see it as over-engineered. These feature aren't flaws though and are actually a good reason to want to consider CTools. I hope our coming review of the CTools plugin system better explains this and the individual parts of the system. And until we have these reviews there really isn't much point in making conjecture about these systems.

Plugin declaration hooks and factories

donquixote's picture

Items #1 and #2 are such perfect fits for a PHP Interface and a PHP Class (respectfully) that it's not worth discussing other alternatives. The rest are still somewhat open.

Nice.
But we still need something like declaration/factory hooks.
And then we need to decide if we invoke these hooks on every request where we need these plugins, or if we store the results somewhere.
In case of factory hooks, any storage between requests would require object serialization.
Otherwise, the plugin system needs to know how to construct objects from raw info data. Ideally with a factory function in between. So the info-style declaration hook would specify a factory name and a bunch of arguments.
Or a serializable factory object?

human-editable JSON files

Ever considered YAML?
Would these files be web-editable? It is always a challenge to let a human and a machine work on the same data..

While I like YAML, I'm

glennpratt's picture

Ever considered YAML?

While I like YAML, I'm guessing JSON gets a boost by being baked into PHP.

Indeed. YAML is currently

sdboyer's picture

Indeed. YAML is currently only available via PECL. That makes it a no-go, as this is a requirement for a super-low-level subsystem.

yaml for php: pecl, syck, symfony

donquixote's picture

For the record...

PECL yaml:
http://www.php.net/manual/de/book.yaml.php
runs at C speed, but I agree it is not something we should expect or require.

syck:
http://pecl.php.net/package/syck
I don't get if this is the same as the pecl yaml above.

symfony yaml:
http://components.symfony-project.org/yaml/
Symfony 2.x uses (probably as a fallback, if no C extension is installed), its own php implementation. They say it's fast, but probably slower than something written in C.

spyc:
http://code.google.com/p/spyc/
Another (older) php implementation. Probably slower than the symfony version, if we believe the symfony guys.

The question is, how big is the performance price we would pay by using one of the php implementations, compared to pecl yaml / syck or json.
One should guess that performance of native json_encode / json_decode is similar to that of pecl yaml, and any php yaml is significantly slower.
Only tests can tell.

Conclusion:
It smells like json is the better option for availability / performance.

EDIT:
I imagine there are more appropriate places for this discussion, where probably it has all been discussed to death already. My apologies for starting it here.

Yes, that would be the

sdboyer's picture

Yes, that would be the verbose version of what glennpratt and I said.

I've done alot of YAML with

mikey_p's picture

I've done alot of YAML with PHP work on some other projects, and I believe that Symfony uses Spyc and I would guess that it's no slower than our current drupal info file format.

When we created .info files,

merlinofchaos's picture

When we created .info files, we considered and rejected yaml because yaml is not really human readable and writable, while .ini format is.

Personally, I think yaml is interesting but not ideal.

If you keep the yaml as flat

donquixote's picture

If you keep the yaml as flat as our info files, then there won't be much of a difference.
Otherwise, I would rather say "more easily human readable than".
But, I don't insist on yaml.

Off-topic

Crell's picture

The configuration file format is off topic for the Butler group. The Configuration Management group is where that would be discussed, and I'm fairly sure they're planning on JSON. We'd just be building on top of whatever it is they do. That's not a decision we have to make.

Most of it requires, given

donquixote's picture

Most of it requires, given Drupal's current design, either a full database load with lots of queries, lots of really big nested arrays that get cached, or hook invocations that therefore require a full bootstrap.

What parts of the bootstrap are we afraid of?
Module file includes? I imagine in D8 we could have a modulename.boot.inc, that would contain hooks that have to run early.
Db initialization? We need the db anyway, to decide which modules are enabled.
Anything else?

Db initialization? We need

glennpratt's picture

Db initialization? We need the db anyway, to decide which modules are enabled.

Perhaps the system table could be replaced with a configuration file...

Lots

Crell's picture

Part of this process, long-term, involves a heavy rewrite of the bootstrap process. We need the systems in place to support that, though.

We want plugins to work as early as possible. That means pre-database, because a number of systems do initialize pre-database and we want to be able to leverage both context and plugins for that. Even something as simple as looking up what class is the handler for a given context item could be expensive if we do it wrong.

To do pretty much anything other than hook_boot right now requires a full bootstrap. That's godawful. One of the things we want to be able to do with this process is do, say, a node_load() and return it as a JSON object while loading only the absolute minimal amount of code necessary. That means full bootstrap goes away. Most of /includes is then utterly useless. Etc.

Moving most of our configuration information to simple disk reads and json_decode() eliminates a lot of problems that right now have no solution other than hard-coding things in $conf in settings.php or a full bootstrap. There's no in-between. The configuration core initiative, hopefully, will give us a more robust system there to leverage.

node_load() and return it as

donquixote's picture

node_load() and return it as a JSON object while loading only the absolute minimal amount of code necessary

We need at least a permission check for node_load(). And this requires to know who is the user.
In current Drupal we would also need to call hook_node_load() (D7) or hook_nodeapi('load') (D6), which in the current implementations can depend a lot on global state that was initialized in hook_init().

What I would suggest is a slim/minimal bootstrap, plus a system of lazy loading for user, permissions and other things.

And please tell me how can we check permissions (for a given user) without touching the database? Store perms in the session?

Global state in hook_init()

catch's picture

Global state in hook_init() will need to be replaced by the context system.

With hook implementations, I'm very hopeful that we can add lazy module loading to module_implements() - so load modules (and their dependencies) on demand, but not before. This is easy with hooks, not so easy with things like hook_menu(), theme registry and other places that have alterable callbacks with no explicit module ownership.

Users can already be cached with entitycache(). We could cache permissions per combination of roles easily enough (I think I had a patch for this somewhere but don't think it made it into D7) - if we cached that, it's the kind of thing that could live in cache_bootstrap and live in APC.

Global state in hook_init()

donquixote's picture

Global state in hook_init() will need to be replaced by the context system.

Yep. And hopefully, a lot of this will be lazy loaded.

alterable callbacks with no explicit module ownership.

I expect that we will have namespaces in D8. This allows to tell the module ownership for any registered callback.

I expect that we will have

sdboyer's picture

I expect that we will have namespaces in D8. This allows to tell the module ownership for any registered callback.

Not the problem catch is raising. He's pointing out that the data produced by hook_menu() implementations (where we do know the owning module, at least most of the time) can be altered in hook_menu_alter(), meaning that by the time the full menu system's data has been assembled, we have no idea where it's all come from.

Yes exactly

catch's picture

This is the same with hook_theme_registry_alter() and others. Although I guess it'd be possible to have the same namespacing for callbacks -my_module__callback_name() and load the module via that.

Still don't get it. A

donquixote's picture

Still don't get it.
A function mymodule\foo (or "mymodule\\foo", wtf), be it a page callback or a theme function or whatever, can easily be identified as belonging to mymodule.

Yes, sorry, IF we have

sdboyer's picture

Yes, sorry, IF we have full-on PHP5.3 namespacing in D8, then it would be possible. But that's a HUGE if - we're not even clear if we'll have a php5.3 dependency, let alone shift over to using namespacing. I'd rather delay the namesapce battle until D9, personally.

If we don't get namespaces,

donquixote's picture

If we don't get namespaces, then at least we need something like a 'hook' or '__' separator to get out of the nameclash fragility. And then in D9 we go namespaces.. Duplicate work. I'd rather see us go straight to namespaces.

It would be more readable and

pounard's picture

It would be more readable and leave some ambiguity we today experience with hook naming, but it'd imply the definitive PHP 5.2 support abandon. But I'm not against progress here :)

Pierre.

Off-topic

Crell's picture

The design for plugins does not need PHP namespaces, and Butler does not include any plans to change the way hooks work. (There may be a good argument to do so, but it's not part of Butler.) Please stay on topic. Revamping the hook system to use namespaces is not on topic.

"Not Invented Here"

RobLoach's picture

Fortunately, a number of people stepped forward to help with that research. Specifically:

  • Sam Boyer and Dmitri Gaskin are researching Symfony 2.
  • Marcus Deglos (Manarth) is researching Kohana.
  • Justin Randall (bejeebus) is researching Zend Framework.

And of course we should be looking at ctools as well. Mind you, I don't foresee us directly using any of the above systems per se. We're not introducing a dependency on Kohana.

Remember that Drupal is terrible when it comes to Not Invented Here syndrome. Do we really need to re-invent something that has already been implemented in at least three other projects? Anyway, just wanted to shoot that out there. I understand that we want to use the best solution available, but it's just something that we should remember.

Exactly

Crell's picture

The reason for researching other existing projects is exactly that: Don't reinvent the conceptual wheel if we can avoid it. If they do something cool, steal it outright. :-)

I don't expect that we'd even be able to leverage significant code from other projects directly, as we are structurally quite different from them. However, if we can leverage their code patterns that saves us thinking time.

I dislike our NIH problem too. We have to address it with baby steps. :-) (Plus, as I said in the Core Conversation I know full well that if I proposed a dependency on Zend or Kohana for Drupal 8 I'd be lynched.)

@Crell Items #1 and #2 are

pounard's picture

@Crell

Items #1 and #2 are such perfect fits for a PHP Interface and a PHP Class (respectfully) that it's not worth discussing other alternatives. The rest are still somewhat open.

I agree so much with this. Relying on the language itself would avoid to developer another CTools, which behind its (maybe) usefull helpers also implement a complex API for handling files, which could be avoided using PHP OOP capabilities and a good autoloader.

Pierre.

Our autoloader is good as it

sdboyer's picture

Our autoloader is good as it is - more efficient than any automatic-filename/filesystem-location-based one because a) we can have more than one interface/class per file which reduces fs hits and may also mean less opcode cache fragmentation and b) ...ugh, can't remember the second point. Larry, what was it?

As of D7, the loading for class-based plugins is pretty well-simplified in CTools (it uses standard autoload, and even does it for you so you don't have to explicitly declare all your class-based plugin files in .info files), but there's still the plugin definition file, all the caching related to that, and of course...all the procedural plugins. Yeah, it's something I've been wanting to improve for years, and while we've made incremental progress on it, ditching functional plugins will help a lot. We'd have some work to do to replicate some of the functionality that's currently in there, but I think it's worth the effort.

Can't imagine Earl will be happy, though. Time to ping him about this...

Your autoloader is so wrong

pounard's picture

The actual D7 autoloader implementation is devoted to failure. It's based on a database registry, which forces the database to be up at really early bootstrap for loading classes. It does I/O over TCP (database) and that's not really that good.

Even when class list is cached, class list can really be a huge junk of data, and goes with PHP huge array performance problems (maybe resolved in earlier PHP, but this existed and may be still true), it's not scalable. With modules such as Views and CTools playing together behind, class list can literally explode in term of size, thus making a huge cache entry, huge database table for code registry, and imply slower PHP array access and exploding memory comsuption.

A namespace/filename based autoader may be a little slower (and that's absolutely not proven anyway) should be way more efficient with a huge code base in the end because it wont iterate over array to find files in it). See lower note.

It would also be a lot more robust, because there would be no need to introspect files and store class list into database. Believe it or not, I tested "DELETE FROM registry" in a D7 instance, the site was dead and totally unable to rebuild it at bootstrap creating WSOD, I had to write my own mock bootstrap manually including the module files for Drupal to be able to rebuild it.

I already spoken of the scalability problem of this autoloader to both Damz and Earl, and also run the idea to Earl that a namespace autoloader could be implemented for CTools and stacked into the SPL autoloader stack therefore would avoid the core autoloader to grow but keeping the compatibility with already existing code. (See lower note also).

There are some good autoloaders out there, Zend_Loader base implementation is a good one. I also did my own for custom needs in some modules, see the http://drupal.org/oox and http://drupal.org/xoxo modules, where a namespace based autoloader lives (use latest git code).

EDIT: More than one class per file is fine as long as you don't need an autoloader, but it's the worst thing you can do for code readability and browsing. It's not non-efficient to use multiple files, most of the PHP framework already took the decision to split up into multiple small file (some took the asumption that an OPCode is installed on the targeted production environment). Even without it, you can rely on the fact that the FS will probably cache most FS lookup, and the first cost a lot, the others remain cached (you should read the linux kernel 2.6.38 version optimizations about this, it's interesting). Drupal is really slow compared to a lot of other PHP applications, and you autoloader I think may improve execution time with misconfigured environments, but will break any OPCode optimizations and will prevent well tuned environment to reach maximum speed capabilities.

Re-EDIT: Some typo errors + This articles are interesting:
http://weierophinney.net/matthew/archives/245-Autoloading-Benchmarks.html
http://www.zyxist.com/en/archives/140
http://framework.zend.com/manual/en/zend.loader.html

I said something wrong, and I'm learning everyday so forgive me, but the map based autoloader seems to be the fastest in benchmarks. That's a point where you are right. I apologize for this detail. I was in the middle of a struggle with my dead site. The highly dynamism in Drupal (which gives great power) also can lead to its load of problems such as in development phase as you cannot switch the autoloader for something else, you are tied to let Drupal parse your code, always. This is one of the point in the second link I gave.

Pierre.

More than one class per file

donquixote's picture

More than one class per file is fine as long as you don't need an autoloader, but it's the worst thing you can do for code readability and browsing.

In most of the one-class-per-file libraries I have seen, such as Zend, Doctrine etc, classes were really huge. In this case I agree we should not have more than one per file.

Also these libraries typically have zero functions.
(I think floating functions are still better / more "honest" than static class methods, singleton and friends)

If your classes are small (that is, one or two 7-liner methods), then I don't see a reason not to put a few of them together in one file, if these classes are logically related and always have to be included together. I hate having tons of open tabs in my text editor.

Could we not have a mix of filename autoload + registry autoload?+

iterate over array to find files in it

Why would you want to iterate?
You load the array from somewhere - either all at once, or separate per module. This does have a O(n) cost, but I think it is acceptable doing this once per request.
Then for every class to load, you look up a key in the array. This is O(log n), not O(n). Arrays are hashes.
Or, for the total request, O(k * log n), where n is the number of available classes, and k is the number of classes loaded in this particular request. k is smaller than n, obviously.

I globally agree with what

pounard's picture

I globally agree with what you are saying here. In fact placing the classes in separate files is more a convention than anything else, but I think it really can help.

I known that PHP array functions are really well optimized, especially isset() which is a language keyword, Damz once learnt me that. Looking up something in a array remains an indexed search. And as I said map based autoloaders seems faster in most benchmarkes (difference is almost nothing in most cases) thanks to those arrays, nevertheless they have a huge disadvantage is that you have to build the map before.

I don't like the code registry, it's like repeating the interpreter/compiler work. I know that people that made it have pretty solid arguments for doing this but it seems a bit obfuscated, while a lot of frameworks attempt to generalize classes and naming conventions to avoid this kind of stuff.

Pierre.

you are tied to let Drupal

donquixote's picture

you are tied to let Drupal parse your code, always.

There are three ways to fill the map:
1. Let Drupal scan/parse your files.
2. Actively tell Drupal about your classes and files.
3. Let Drupal guess based on filenames.

I think it would be possible to allow all three methods. The file extension (*.class, *.inc, whatever) and module subfolder name would tell Drupal to use one of the three methods for this particular file.

Sorry, I should clarify -

sdboyer's picture

Sorry, I should clarify - when I said "the autoloader is good as it is," I should have restricted the scope to how we structure the files & locate them on the filesystem: as many classes/interfaces as you want per file with an arbitrary name, and anywhere you want for the file. The benefit here is

and also run the idea to Earl that a namespace autoloader could be implemented for CTools and stacked into the SPL autoloader stack therefore would avoid the core autoloader to grow but keeping the compatibility with already existing code.

When I wrote ctools integration with the 7.x core registry, I considered this approach and rejected it because it ultimately would have just resulted in us needing to hit more db-backed caches or do filesystem scans to locate the class. There are no naming restrictions whatsoever on classes used by ctools plugins - there aren't even conventions, really. So there's no way to be smarter about it. At least, not that I could think of - if you have an alternative that is compatible with existing code, I'll consider a patch.

The actual D7 autoloader implementation is devoted to failure. It's based on a database registry, which forces the database to be up at really early bootstrap for loading classes. It does I/O over TCP (database) and that's not really that good.

It would also be a lot more robust, because there would be no need to introspect files and store class list into database. Believe it or not, I tested "DELETE FROM registry" in a D7 instance, the site was dead and totally unable to rebuild it at bootstrap creating WSOD, I had to write my own mock bootstrap manually including the module files for Drupal to be able to rebuild it.

Yeah, this is something that definitely, definitely sucks about the D7 registry (and wasn't what I meant to refer to in my post, but is quite valid). With the addition of a config API, though, we could help the problem by dumping JSON to disk containing a) the compiled list of files for introspection and/or simply b) the entirety of the autoload list, k/v pairs of classname/path-to-file. At that point, it becomes a caching problem like most others - just let the site admin pick where they want to pull the list from.

Oh I see, we agree on a lot

pounard's picture

Oh I see, we agree on a lot of things then. I now that finding a naming convention seems simple, but porting the existing code is not. There always a simple solution to keep the compatibilty, is to keep the actual autoloader (deprecating it or not) while making it run side by side with another one. SPL allow autoloader chaining, even if some framework prefered to implement their own autoloader stack (such as Zend, because they couldn't reorder the SPL autoloader stack like they wanted, the function seems not to always do what you want it to do).

Pierre.

On autoloaders

Crell's picture

As pounard's later edits noted, benchmarks show that an array lookup autoloader is faster than a string parse autoloader.

Drupal is simply incompatible with a string-parse 1:1 file/class setup anyway. I discussed that at length with the PHP Standards Group over a year ago:

http://groups.google.com/group/php-standards/browse_thread/thread/23e99c...

We could not switch to that approach globally without rewriting essentially the entire system around it, and there wouldn't really be much benefit from doing so. However, it is possible that we may be able to do that for selected core parts of the system that need to initialize before we have access to either the database OR a disk-based config system. Say, portions of /includes may benefit from moving to that structure as an optimization so that no meta-knowledge is needed at all to load them. We need to be careful about include() misses, though, as those can be expensive. That would take more research.

That said... this is not the place for such research. :-) Using classes for plugins means we can use an autoloader. Period. The exact nature of Drupal's autoloader is a Drupal-wide question, and not something relevant here at this point. I am not saying that our current autoloader cannot be improved, it can, but please stay on topic and improve the autoloader in a separate process.

This thread has already been bikeshedded to death with off topic concepts. I mentioned the JSON config file stuff as an after-thought. This thread was supposed to be for research results. I've now opened a new thread for that. Please, everyone, stay on topic. We have a ton of work to do and getting stuck in analysis paralysis is not going to help anyone.

Agree, let's go back to the

pounard's picture

Agree, let's go back to the original topic, sorry for disturbing.

Pierre.

OOP and Ctools plugins

neclimdul's picture

You're absolutely right, OOP does handle those 2 points quite well and a good autoloader is quite handy. That's not mutually exclusive from using CTools plugins as you seem to imply. File handling is not the only thing CTools plugins do. In fact, CTools plugins in d7 quite nicely integrate with the autoloader meaning that file handling your talking about just populates the core auto-loader instead of having to define them in your .info. You can thank @sdboyer for a lot of that work btw. :)

What CTools plugins is actually about is not file handling, that's just an important side effect. They are really about managing meta information about the plugins in a consistent and maintainable way. Just tossing some interfaces and classes together doesn't solve the problem of populating select lists in an admin interface with human readable names, providing a peered theme function and a test url and callback function all managed neatly in a directory structure and set of includes. That is what CTools plugins are about.

That's just the sort of thing you need for #4, #6, #7(you've got to define your ability to be cached somehow), and #8 along with other use cases people using CTools have come up with over the years.

I just wanna back this up -

sdboyer's picture

I just wanna back this up - we (me, merlinofchaos, neclimdul) argued about this for a while this morning, and I was reminded that though the core functionality of a plugin is well-handled by interfaces & OO, there's a lot to do to be able to drive them to the UI. A Drupal UI. I'd not been considering this recently, but it's not been given nearly as much attention as it should. Listen to people who've been doing it in Drupal for years; you do NOT want to give UI representations of plugins short shrift. Plugin user interfaces are at least as important as the machine interfaces.

Agree with the UI problem,

pounard's picture

Agree with the UI problem, this is something that should been take care of carefully, but it shouldn't impact the normal runtime.

Pierre.

Agreed, but separate

Crell's picture

On one level I agree. We do need to have a consistent way to handle UI components for plugins. My current thinking there is to have form methods in each plugin's configuration object that essentially mirror what Views plugins do now (options_definition(), options_form(), etc.). That's a fairly good model, I think, but should be separated out into the config object. We do want to solve that problem generally, as it makes building UIs for pluggable stuff easier. I was rather hoping I could talk you into being involved in that part, precisely because I know ctools has dealt with a lot of these issues already. :-)

On the other hand, we actively want to avoid making plugins dependent on their UI. The API for manipulating plugins from code must come first so that it's a real API, and then a flexible GUI can be built that leverages that API. Assuming a 1:1 mapping from UI to data structure is one of the dumbest things Drupal has ever done (Node API, CCK Drupal 5, etc.) and yet it does it over and over again. Let's not make that same mistake again. We don't need more naked objects.

I implemented another

pounard's picture

I implemented another solution in some custom modules (which has its own flaws as well):
I create a "FormableInterface" with three methods formBuild($form, &$form_state), formValidate( ..) and formSubmit(..). The goal of the submit handler is to modify the object itself, of course.
So you can do pretty big forms based on inheritance, or make pretty huge forms mixing up on composition using multiple objets' forms together.
Another solution would derivate this by implementing a function such as MyPluginObject::getFormHelper() which would return such object.

I like the admin UI being exported to config objects, it depends on how the config object definition ends up. I mean if config objects are pretty much how I described the formHelper, it seems a composition pattern, so config specific object implementation can be used and/or extended to be share with multiple plugins.

Pierre.

Not even talking about forms.

sdboyer's picture

Not even talking about forms. They're important, but what I'm talking about something different. Solve this problem, for Drupal, given the constraints:

Populate a select list element with a human-readable list of plugin names, keyed on an id that uniquely identifies a plugin.

The fact that CTools handles

pounard's picture

The fact that CTools handles metadata for those plugins is probably an anti-pattern. OO objects with a solid interface based design describes most of the needed data by itself.

By managing plugin metadata and doing a lot of runtime check you probably are trying to substitute the PHP interpreter and OPCode cache optimizations per your own creating a new risk that you end up with implementing anti-optimization and dawn slow code.

If plugins switches to full OO, the only metadata you probably need, at the begining, is the fact they exists (a plugin registry) no less, no more. By constently chekink in userspace memory metadata by looping into huge string array, you'll probably end up with really dawn slow code.

Pierre.

Much the same argument I made

sdboyer's picture

Much the same argument I made - there's no good reason to do in userspace what can be done in language-native constructs. The point about UI elements remains, though - it's quite a bit quicker to iterate over a large set of plugin arrays and snag a value than it is to instanciate reflection objects for each class then call $refl->getProperty(). It wouldn't be too bad for less frequently used edit-time operations, but if it starts creeping into view-time...ugh.

re: reflection instanciation, i've done some benchmarks: http://blog.samboyer.org/blog/microbenchmarking-php-performant-code-now-... .

Instantiation cost

Crell's picture

I disagree entirely. While you certainly can build a system that exposes metadata via methods on the class, that means that to get any metadata at all you need to do two things:

1) Parse the file the class is in into memory, taking up RAM that you can never release.
2) Instantiate the object, which means passing it whatever it needs for its constructor even if it makes no sense in context.

Using an opcode cache helps with the first, but it does not eliminate that cost. It also does nothing for the latter. We need to know things about a plugin without parsing its code or instantiating it. If we had to parse all pugin classes and instantiate them to get information about what they're for, we would blow out our memory limit on even an opcode cached-site, to say nothing of a normal site.

What we really want here is string-parsable annotations. However, there is no such system in PHP at present, and discussions about adding them to the language of late have fallen flat. It certainly won't be happening in the timeframe we need. That leaves us two options for metadata:

1) Info hooks.
2) DocBlock-based annotations that we strong-parse for.

Between those two, we collectively have far far more experience with the first and know its ups and downs well. Someone would have to make a very strong argument with sample code for the latter for us to switch to that, although I suppose I am still open to it if someone could provide a working proposal.

Also, I want to make one thing extremely clear: Our target platform is a non-opcode-cache site. In every single benchmark I've run or seen or even heard of, all of the talk of memory page fragmentation or cache performance or other esoteria amounts to maybe a few percent difference total. It doesn't actually mean squat. However, Drupal 7 does not work without an opcode cache. Just this week at work, we concluded that a perfectly good shared host we'd been using for years cannot run Drupal 7 at all because the modules page dies on its 64 MB limit. 64 MEGABYTES of memory, most of which is simply loading code off disk we don't use. And that's with only a scant few modules installed.

Assuming that an opcode cache will fix our problems is why Drupal 7's performance is utter crap. We are not making that mistake again, at least not in this core initiative. We must reduce our code weight and load far far less code per page request than we do now.

A no-opcode-cache server that is not over-sold and is run by a vaguely competent hosting company is our target environment. Please focus on that configuration when making arguments about code structure.

Thoughts

eaton's picture

Assuming that an opcode cache will fix our problems is why Drupal 7's performance is utter crap. We are not making that mistake again, at least not in this core initiative. We must reduce our code weight and load far far less code per page request than we do now.

A no-opcode-cache server that is not over-sold and is run by a vaguely competent hosting company is our target environment. Please focus on that configuration when making arguments about code structure.

If I were not married, I would move to Canada and gay-marry Crell right now.

Just because this post.

+1

walkah's picture

If I were not married, I would move to Canada and gay-marry Crell right now.

Just because this post.

I endorse this move and will sponsor you for immigration.

Err, no

catch's picture

The reason Drupal 7 performance is shit is because thousands of patches were committed that didn't give a second thought to performance implications, many of them causing significant regressions that had to be found manually later (and usually by me). Even some patches which did give thought to performance implications, like the registry itself, had issues that were not found until a long time after commit.

This statement is true regardless of whether you run Drupal 7 with an opcode cache or not - either way there is a significant regression for both CPU and memory usage from Drupal 6.

The question then, is whether the regression from Drupal 6 is so much worse for non-opcode hosts than it is for opcode hosts. I don't think that's the case currently, for various reasons that are too in-depth to go into, and off-topic for this thread, but which I wish people would spend some energy researching and discussing instead of repeating and cheerleading received wisdom here.

Today with D7, running

pounard's picture

Today with D7, running without OPCode caches seems to be pretty much a suicidal thing to do. But whatever metadata you need, OPCode remains basically pure optimization, but some stuff like eval'd code pretty much break those.

As I say, it depends on the metadata you really need. Basically, if your plugin is nothing more than a specific implementation of a specific interface, I don't see what kind of metadata you'd want. If you are talking about a human name or human description for administration screens, then yes, there is a real use case here of useful metadata, but it's nothing more like simple hook_something_info() and a nice plugin registry would handle this.
But for running code, once your objects are spawned, they don't need all this to run, they just do whatever they have to do.

Please, don't talk to me about parsing the code, I would never run PHP parsing itself (seems a bit obfuscated to do that).

If your script eats something like 64M of memory, OPcode cached or not this will remain 64M. OPcode cached code doesn't mean code that eats less memory. So in that sens I agree on the fact that targeted environment may be non-opcode ones, because with or without, it won't change the memory consumption at all, and fast code on non opcode env is fast code with.

Pierre.

Coming in late

Crell's picture

Pounard, I think part of the disconnect here is that what is now being called "Butler" has been in the works for well over a year, so you're re-tredding subjects that have already been over, both in this group and elsewhere and in code. You talk about using hook_something_info() for handling the metadata; that's the obvious conclusion that I reached about 3 years ago, and Earl Miles did about 4 years ago. Until/unless we have language-level annotations I don't see any other viable alternative to info hooks for that sort of thing.

By "parsing", I don't mean PHP's user-space tokenizer (although there's nothing inherently evil about that) but simply calling include_once() on a file. That ends up using memory that never gets released, in addition to the CPU time required.

An opcode cache helps with the CPU usage there, certainly. It also, in my experience, does lower memory usage as different processes can share the parsed code. I'm not sure of the details but I have seen memory usage go down from installing an opcode cache. Again, though, all of that is irrelevant as our target system has no opcode cache.

Let's move on and say this is

pounard's picture

Let's move on and say this is off-topic, this discussion could continue days and days.
To advance over butler, see my latest posts (excluding this one).

EDIT: I'm not rethreading in code, I did ask you questions about your issues. I read your code, tested it, and asked questions about it, had some fun time writing mine and given it to you just as is. Not to say your's isn't good, on the opposite, but because there wasn't no better place to put it.

And my code may rethread your issues, it remains in the bulter scope given the post I read (ESI and cache problems, request and response objects, hierarchical MVC with a dispatcher, multiple controller, context inheritance and such fun stuff described in so many post in this group).

Pierre.

Not irrelevant at all

catch's picture

Our target system may or may not have an opcode cache, eventually APC will be included in PHP core, so it could end up being more often than not.

Also there are costs to including files (i/o, memory and cpu) which occur whether or not you have an opcode cache enabled, or depending on how that opcode cache is configured - these costs are on top of the things that opcode caches optimize, but they become more important once your server/site is already well optimized and should not be ignored.

The original post gives a

pounard's picture

The original post gives a good question, how to handle plugins? But I think there is no general answer. The fact that, for cache backend for example, it can work without any UI here, it's basically not something you configure (you do, but not through the UI), once this configuration is set up, you are unlikely to change it except if your environment needs it.

I see at least two types of "plugins" here:

  • Low level components, that are determined at run time, or by the sysadmin, depending on the environment or some particular environment or site specific stuff. Cache backends for example is not something that you'd let your site integrator configure, because the person who make the site is probably not the one that installs it. Therefore this kind of components metadata is only documentation, and their UI should probably remain in pure configuration files (such as settings.php).

  • High level components (let's say, to avoid speaking of CTools here, for example Feeds' module node processors), these are components into a high level specific object tuple that is meant to accomplish a complex business task (Views is a good example also). These needs a good UI, and a good registry for being able to hanle them directly in UI.

So I'd say we should probably not rush into CTools model, CTools handle problem in an historic fashion, because it was primarly meant to a specific use case (procedural plugins) and I think now, even if it works and it solves some problems, this model is outdated.

I like what Crell proposed about having "configuration objects", but I think this kind of objects are meant to be used as helpers for UI, once an specific component is good and running, it doesn't need it anymore. Things such as the Views' options handling implies that you do not make direct object's property access but options array instead, and this kinda sux in a way is that the component developer is stuck with this model. Configuration objects remains a really good generalization for data normalization and storage, they probably would basically be mappings objects that would allow easy import/export/storage operations.

Something that CTools and Views does that sux is storing objects into database. When you edit a Views over time, some of these options will remain into the object and you may end up with a lot of unncessary garbage being kept (Views does this all the time) and their export model by doing direct objects var_export() keep this garbage, and canno't be easily serialiazed using other formalisms (JSON, XML, etc). Using a mapping layer between the objects and the storage (I don't know if you meant this aerlier, but for me it would be a good usage of configuration objects) would totally improve all of this.

Pierre.

context, plugin, handler definitions

drifter's picture

Could we have a seperate page with definitions of what a context, a plugin and a handler mean in terms of the Butler project? I have an inkling, but since these have been used in so many contexts (pardon the pun) in the past, it would be great if there could be definitions we could refer to.

Already asked this, a context

pounard's picture

Already asked this, a context remains undefined to me. I see a context as a "business context" linked to some modules whatsoever (for example, an OG group), derivated from input data (such as the GET parameters or current path) at application init (after bootstrap, before running the menu item callback) globally visible for a given scope (a controller action run in the MVC model).

But I might be wrong.

Pierre.

I'm not sure business context

neclimdul's picture

I'm not sure business context would be the right term. Its any information that provides context. The node being rendered, the current language, the current groups, the current user, the author, some global site state, etc. If you consider a couple of these being on a context object and look at the diagram in larrys OP you can see how different block could respond to this information without sideloading and the information from some global space such as a function, the path, or a GET argument.

If you includes things such

pounard's picture

If you includes things such as user, current theme, globally framework and pure technical information, I agree this is not business stuff. In the other hand, if you get stuff like current node being displayed, OG group, or any other kind of data the user manipulates, then it's business stuff. Language can be considered either as purely technical information (typically interface language) either as business information (current node language for example).

I'm pretty sure that both should be distinguised, pure technical related information is static and does not depend on any installed module (not even fields or entity), let's says that this information is determined at bootstrap time and should remain static over the full execution flow, while node in the other hand belong to a module, it's something the core framework doesn't now until the node module is enabled (even if the node module is mandatory, it remains a module).

These information can be treated the same way, I mean if I need to compute a serializable and reversible GET URL for an ESI gate, in order for the framework to rebuild the current full context and render only one single piece, you need it to be stored at the same level. Nevertheless, in this particular case, static and technical information shouldn't be overridable while business stuff can be.
You can differenciate them in the implementation, basically the technical information can be regrouped under something that is a global context which in any way won't be overrided by modules, while business context could be at every level of the execution flow: each "controller" could add, override or remove contextual information it (doesn't) need.

But, in a more general way, some technical information could be see as business stuff (current logged in user for example).

I did some pieces of code in order to be able predictible URL for an ESI module: http://drupal.org/project/esi_api (actually more than a proof of concept than a module), and this where I encounter some problem: In this module, I manipulate a variable array, which actually is a context. Some of the information from this context comes from the GET, some others from core configuration, and some other from the business function (such as the block rendering function) and all of this is sanely merge in order to compute a full predictable query string that allows me to reproduce the exact same context when hitting the ESI callback. In order to make it efficient and ensure a lot of cache hits, I had to manually remove some of the technical information and such using some predicates such as "not all information is useful to rebuild a context" and some other like "theme key shouldn't get to the final query string if current being used is the default one" and stuff like that.

In order to be able to this gracefully I had to differenciate business context and purely technical context, if I hadn't I wouldn't have been able to build query strings with minimal information (the minima is often what need the business callback to rebuild its content).

Pierre.

Posted above about Context

neclimdul's picture

Posted above about Context (an object containing relevant context such as a node or language).
Plugins are replacable bits of code. They are generally classes but ctools is cool with them being functions. They are seperarate from modules in that there may be multiple implementations per module. Think input format, cache backend, block, panels layout, view filter/field/argument/style.
Handlers refers to a plugin system with routing logic and configuration for handling a situation. Context is important to this for routing the plugin and its sort of a synthesis of all these concepts.

That's where I do not agree

pounard's picture

That's where I do not agree with the plugin definition you gave.
"A plugin is a component that respond to an interface." is probably more like it.

A component is code, an interface is not mandatory an OO interface (modules that respond to hooks are basically plugins, hook definition is the interface, while hook implementation make the module being a component responding to this interface).

What Crell intend to do, if I'm not wrong, is to give an interface for interfacing, which is quite weird to get to this level of abstraction, but totally understandable, but by doing this it also restrict the global plugin uses cases to only the one that fit with this abstraction level.

I see them more like: storable, formalisable and configurable components that may or may not need a UI. In OOP programming, a pure plugin is a class that implement an interface, no more. In Drupal, for UI needs (and such), those classes, that implements interfaces needs to be registered globally, so the UI can be built over and let the user either choose its implementation for a specific business stuff, either configure the components.

EDIT: I do not agree with "Plugins are replacable bits of code" either. Replacable bits of code denotes dependency injection or the adapter pattern, which is only one the thousand use case a plugin can have. For example, a Views' style plugin uses the adapter pattern, it can only be one then yes, it's a "replacable bit of code", but Views' filters are not, it might exist some kind of adapter pattern here (you can't put a database filter if you are querying SolR) but this is not dependency injection because you can have a lot of them, and you don't need to replace them at runtime depending on the environment.

Pierre.

not an adapter

donquixote's picture

Replacable bits of code denotes dependency injection or the adapter pattern, which is only one the thousand use case a plugin can have.

Eh what?

An adapter is the thing in between, that allows to components to talk if their interface does not match. I don't see what this has to do with plugins or "replaceable bits of code". Obviously a plugin can be an adapter/wrapper/facade, that hides/wraps one or more other objects, but it can also contain the implementation by itself.

For Dpi I think we rather should start a new thread. But in short, I would summarize, firstly, that you get your dependencies (which can be objects or just any data) injected as constructor or function arguments, so you don't have to pull them from global state or construct them within the component. In Fowler's definition, it also means that some external component will look at the class and see what object types it needs to inject - in php this typically translates into reflection.

Again, I don't see how dpi would be a "use case" of a plugin. It simply describes the way to make the dependency (which can be a "plugin") available to the consumer component.

I think the definition of "plugin" should be discussed in another thread.

Already determined

Crell's picture

We've already determined quite some time ago that a plugin will get a context object and configuration object injected into it in the constructor (and possibly other metadata). Ideally most if not all of the information a plugin needs will come from those two sources, but at the moment I cannot say that is an absolute. (Nor could we really enforce it for arbitrary plugins.)

I'm right smack in the middle of moving, but once I'm settled I'll try to put together a canonical list of "knowns" and "solved problems" so that we don't rehash existing topics more than necessary.

Most of this thread at this point is off topic, so I've kinda given up on bringing it back on track. :-(

Not that off-topic, I lacks a

pounard's picture

Not that off-topic, it lacks a good plugin definition (until I saw donquixote note which is actually the best summary I saw of the plugins problem. All these discussion, IMHO, denotes two facts:
* The right words are not being used, it confuses people that didn't took part of live or earlier discussions
* These plugins and context stuff are really heavily coupled with the other butler problematics (having a better (H)MVC pattern, ESI and cache oriented rendering, etc..) and the whole architecture can only be determined either by cutting down to even smaller pieces (such as: how to implement a generic plugin registry would be a good place to start) or a higher/wider architecture point of view (how and where do you connect plugins with the rest of the global new Drupal architecture).
I think the off-topic post well express that. The question on the original post is too large to be direct, and too open not to spread.

EDIT: Some typo errors (sorry for multiple mails, put in place good mail filters :)

Pierre.

Again, I don't see how dpi

pounard's picture

Again, I don't see how dpi would be a "use case" of a plugin. It simply describes the way to make the dependency (which can be a "plugin") available to the consumer component.

I was a but abusive speaking of the adapter pattern, I always have too much in mind use cases such as database adapters. "Replacable bits of code" make me think of that because when using the adapter pattern you often are in the case in which you can have only one active component at a time to so a single task. Dependency injection is just a word to express the need for a part of an API to be plugable (often uses interfaces and components that implements them) you can switch depending on the environment (or any other condition), basically every component loosly coupled implementing an interface is subject to be used somewhere injection dependency is possible.

If I read well "replaceable" means "that can be switched" for something else, I just quoted two very frequent use cases.

A plugin is not always "replaceable". A lot of plugins that create new features are not meant to be "replaced" they are meant to be activated or not.

Pierre.

Routing unnecessary?

eaton's picture

A way to define logic for "find me the right plugin for X circumstances". (eg, "cache bin for pages". I've been calling this routing.)

I'm still not convinced that baking this system into the plugin architecture is necessary. As long as plugins are capable of delegating to each other, making 'wrapper plugins' that handle routing internally using configuration settings is a reasonable approach. Very, very few of the plugin use cases that I see really need any kind of routing functionality, and this adds a lot of complexity to the system in the form of 'generic configuration'.

First, let's just never ever

sdboyer's picture

First, let's just never ever call this "routing" again. Sorry Larry, but nobody thinks of "routing" being a good word for this, because routing has a lot of other meaning attached to it.

Second, I'd like to default plugin selection baked in as an optional piece usable by plugin types - if it makes sense for that plugin type. There are half a dozen cases in VCAPI, in use on d.o, where we could benefit from the system itself providing a default when none is explicitly specified. Since no such system currently exists, we wrote our own hacky one.

Did some bits of code for fun

pounard's picture

I did some bits of code for fun, you can check out this:
http://drupalcode.org/sandbox/pounard/1109806.git/tree/897e54b:/pluginre... (from the http://drupal.org/sandbox/pounard/1109806 sandbox)

Actually inspired from Zope plugin registry, with some bits of totally drupalistic code, based on some hook_info() with a working use case. Taking account of some stuff such as registry is also a factory, a single entry point (static registry for factories) and some goodies such as null object pattern basic implementation and stuff.

Pierre.

Improvements to core

Group categories

Category

Group notifications

This group offers an RSS feed. Or subscribe to these personalized, sitewide feeds:

Hot content this week