WSCCI Plugin Sprint Summary

At DrupalCamp Austin, several people from the WSCCI and CMI initiatives as well as the ctools team met to discuss the topic of Plugins. Much progress was made, and we were able to develop a plan for several still-TBD issues.

Attending

Larry Garfield - Crell
James Gilliland - neclimdul
Kris Vanderwater - EclipseGc
David Strauss - davidstrauss
Michael Favia - michaelfavia
Kurt Vanderwater - kvanderw (scribe)

Slogan

"The goal of the plugin system is to get everyone into the same chapter, not necessarily onto the same page." --David Strauss

Background

The "plugin question" has plagued Drupal for some time. We have hooks, which are a great way to respond to events within the system, and to add functionality to an existing system. They are a rather poor way to define "swappable components", however. Currently, core provides separate, inconsistent mechanisms for such components for every system that needs them: cache, session, text formats, blocks, password hashing, and various other core systems all have their own distinct one-off, inconsistent way of defining swappable stuff. Not only does that make it harder to learn, it means we can get no optimizations from common patterns or common code, because there are no common patterns or common code.

Various contributed modules have swappable systems, too. Most are similarly one-offs of limited capability. The most developed, and most widely used, is the ctools plugin system. Conceptually it is quite robust, although its implementation now carries around a lot of legacy baggage that make it more complex than we would want for core.

By providing a unified plugin API for core, drawing heavily on ctools plugins, we can simplify Drupal's APIs, improve learnability, and improve performance by having less code and a common way to lazy-load code on demand, and provide more power for module developers.

We previously investigated a few other PHP frameworks, and found that none of them had plugin systems as robust as we needed. However, the underlying concepts are seen in many other projects, including Symfony2, many Java programs, and the Cassandra document database.

Why does Web Services care?

So why are plugins relevant to the Web Services and Context Core Initiative? There are two primary reasons:

1. Part of WSCCI is implementing a context-aware block-centric layout mechanism that allows individual blocks to be configured, placed, and rendered independently of the rest of the page. That is necessary for more robust caching and layered rendering. Doing that, however, creates another use case for swappable components. Rather than implement yet another one off, the WSCCI team feels strongly that it is better to build a common plugin system that we can leverage alongside the rest of Drupal.
2. In order to return web service requests in a performant way, we need to greatly reduce Drupal's code weight. The cost of returning a simple request, such as an auto-complete callback, needs to be low. The easiest way to do that is to convert more code to object-based plugins that are able to lazy-load on demand. In any given request the vast majority of code in Drupal is not used, which means it should not even be loaded. Lazy-loaded plugins that reduce Drupal's code weight help with performance for all requests but especially for lightweight service-style non-HTML responses.

What does the architecture look like?

The architecture for the plugin system draws heavily from ctools conceptually, with some cleanup.

Plugin Type

A plugin type is the grouping for all plugins that satisfy a particular purpose. All plugins are the members of exactly one plugin type and should be interchangeable with other plugin of the same type. The canonical example here is caching. "Cache storage system" is a plugin type. "Database cache", "Memcache cache", etc. are plugins that conform to that type. A Plugin Type must define a PHP interface to which all Plugin Implementations must conform.

Plugin Factory

A Plugin Factory is responsible for discovering available Plugin Implementations and for instantiating a Plugin Instance. A Plugin Factory is a class. The vast majority of implementations, including all core implementations, will use configuration files as defined by the Configuration Management Initiative. Every Plugin Type will use one (1) Plugin Factory, but multiple Plugin Types may use the same Plugin Factory.

Plugin Implementation

A Plugin Implementation, or simply "Plugin", is a self-contained, encapsulated piece of code that satisfies the requirements of a Plugin Type. Some plugins may be used individually, such as cache plugins. Others may be joined together in series, such as text formats. Still others may be assembled into large, complex systems, such as Views plugins. In all cases, Plugin Implementations represent encapsulated logic, and should not be used for data storage. To a system that is using a plugin, its implementation should be (reasonably) opaque. All Plugin Implementations are implemented as a PHP class that implements the PHP interface for the corresponding Plugin Type.

Plugin Instance

A Plugin Instance is a particular configured plugin. A Plugin Instance consists of a machine name, a Plugin Implementation (specifically the class for that implementation), and some configuration. When loaded, a Plugin Instance is a PHP object of the class type of its Plugin Implementation.

Plugin Mapper

The Plugin Mapper is a front-facing object that is the main point of contact for the rest of the system. Module code will access the Plugin Manager and ask it to return a Plugin Instance for a given Plugin Type, with given selection criteria. The Factory will use the selection criteria to decide which Plugin Instance is appropriate to return. The format of the selection criteria is specific to a given Factory.

When loading a plugin, client code will call the mapper with the scope, type, and appropriate selection options (probably an array). The mapper will instantiate the factory for that plugin type, unless it has it already in which case it will be reused. The factory will then use the selection options plus whatever configuration is stored to locate the appropriate plugin object, instantiate it, and return it. In some cases the factory may keep a reference to the plugin and reuse it next time rather than creating a new one, such as for caching. The client code can then do whatever it was going to do with the plugin.

Reasoning

The Plugin Type, Plugin Implementation, and Plugin Instance are conceptually borrowed directly from ctools, and from common object-oriented patterns. It is a structure that is well-proven in the wild, both within Drupal and not.

Although many systems will have factory logic specific to their use case, there are many generic use cases that apply to many different Plugin Types. Forcing every Plugin Type to completely define its own factory logic from scratch would be wasteful. By making it a standard class and interface, however, we can allow different Plugin Types to share factories where it makes sense or subclass and customize them if appropriate. That gives the best balance between flexibility and consistency.

Discovery

One of the big challenges of the plugin system is identifying all available plugin types and plugin implementations. Traditionally, Drupal has used info hooks for that case. However, info hooks have a number of drawbacks:

1. Like any other hook, info hooks are only available after the module system has been initialized, which in turn requires the database system, the cache system, etc. We want to use plugins for lower-level systems.
2. Info hooks are PHP code, and therefore must be parsed into memory to be read. Once parsed, however, they cannot be taken out of memory for the remainder of the request. For plugin types that have a very large number of implementations (blocks, field types, access plugins, etc.), that can be very bad for memory. (We run into this issue now with, for example, hook_schema()).
3. The information from info hooks must be stored somewhere to avoid reinvoking all hooks every time we need to know something about plugins. The normal "somewhere" is the cache system, but we want to use plugins for the cache system. So we'd need another somewhere.

For that reason, we decided that info hooks, while a known pattern, simply would not work in this case.

After some consideration, we concluded that the best place to put the Plugin Type and Plugin Implementation definitions is in the Configuration system being developed by the Configuration Management Initiative (CMI). For background, CMI is developing a system that stores complex configuration in one or more XML files that ship with a module. When a module is enabled, those configuration files are copied into site configuration directory. For more information see the writeup Greg Dunlap wrote after the BADCamp CMI sprint.

As a benefit, these XML files are already slated to become a major part of Drupal 8. That means developer are going to need to become familiar with them anyway for defining module configuration defaults, so it's only one new thing for module developers to learn. It also means that, although there is no alter hook, it is possible to customize a definition for a particular site if necessary through functionality CMI will already be providing.

There is some concern that plugin implementations, since they are not changed by a user or site administrator, do not qualify as "configuration" and therefore should not use the configuration system, even if it is a convenient optimization. An alternative storage mechanism for non-configuration "random stuff" is still in the hand-wavy stage, though, so until that's resolved it's probably best to move forward with using the configuration system.

Each Plugin Type is defined by a configuration file in the form plugin.type.$scope.$type.xml. That file includes the type's human-friendly name and description, the name of the PHP interface that defines the type's API, the name of the class that is the factory for that Plugin Type, and any configuration appropriate for the factory.

Each Plugin Implementation is defined by a configuration file in the form plugin.implementation.$scope.$type.$implementation.xml. That file includes the implementation's human-friendly name and description, the name of the class that is the implementation, and potentially other information as deemed necessary during implementation. That will likely also include default values for configuration options.

Each Plugin Instance is defined by a configuration file in the form plugin.instance.$scope.$type.$machine_name.xml. That file includes the name of the implementation that it is for and whatever instance-specific configuration is necessary.

The $scope part of the above definition is borrowed from ctools, and is intended to avoid namespace collision. For instance, core has a caching system (obviously). Views has a cache plugin system, the implementations of which are highly Views-specific and deal with cache clearing logic, not with data storage. Panels has cache plugins, too, dealing with pane caching, which again are totally different from either core's caching system or Views' caching logic plugins. ctools therefore namespaces plugin types by module name. For core plugins, we have generalized that to "scope", which is a machine-name-friendly string. That will usually correspond to a module name, but that is not required. Non-module plugin types (core caching, for instance), will have a scope of "core".

Another advantage of using the configuration files as above is that enumerating available plugin types, plugin implementations, and defined instances becomes very simple through functionality the Config system is already slated to support, and can be done without loading any additional PHP code into memory.

Factory types

Different plugin types will have different needs when it comes to their loading logic. Sometimes only a single instance of a plugin will be needed throughout the entire request; other times there will be a complex routine determining which plugin is needed. Sometimes, for instance with caching backends, the same exact object should be returned every time. In other cases, such as image processing, a new object will be wanted each time. Etc. For that reason, every Plugin Type may have its own factory, although reuse of factories and subclassing should be extremely common.

Because the definition file for a Plugin Type includes the factory class and its configuration, and configuration files will be copied from the providing module to a site-specific configuration directly, it will be possible to simply reuse the same configuration file to store mapping data. That will be up to the factory implementation.

Although hypothetically it will be possible for a factory to not use configuration files for its mapping data and plugin implementation and instance records, that is not recommended. At the sprint we agreed that not doing so constituted "maiming kittens".

The responsibility of a factory is to instantiate and return a configured plugin instance object. The factory may, at its discretion, save configuration for a particular plugin instance in the plugin.type.$scope.$type.xml file or break it up into plugin.instance.$scope.$type.$machine_name.xml files.

Common factories that core should provide include:

Simple

This factory is used when there is only a single instance appropriate at any given time, such as the password hashing system. It will store all configuration in the plugin.type.$scope.$type.xml file.

Mapped

This factory is used when there will be multiple instances appropriate mapped one to one to a particular key, such as the cache system and cache bins. It will store all configuration in the plugin.type.$scope.$type.xml file.

Direct

This factory is used when plugin instances will generally be loaded by machine name rather than by a configured map. The standard example here is blocks. It will not make any modifications to the plugin.type.$scope.$type.xml file, but will store every instance in its own plugin.instance.$scope.$type.$machine_name.xml file.

Compound

This factory is used when a given plugin represents multiple internal plugins, such as the text formatter system. We're not sure yet how it will store data.

More complex systems, such as Views, will likely implement their own factory extending the Compound factory.

Context and Configuration

One somewhat contentious question in the past has been whether or not plugins should require context objects, and whether all plugins should require a configuration object. There will be many plugin types, or at least implementations, that do not require one or both of those, but having different flavors of plugin that take different dependencies would make the system more complex. However, making all plugins require both of those means that plugins cannot be used before both of those systems have been initialized.

After some discussion, we reached the following conclusion:

1. Plugins will be available really really early, but have to be available after the configuration system because everything else in them, such as the factory logic, requires configuration.
2. Plugins that do not require configuration can easily be passed an empty configuration object, so no harm done.
3. The Context API should be available very early as well. In fact, the only systems that might need plugins that happen before the Context API is initialized are Context itself and Session handling. However, the Context API is already mostly implemented using a different approach to handler-mapping. We also have been looking to implement lazy session initialization via the Context API, and rather than use our own plugin system for it rely on the session storage classes provided by the HttpFoundation library, which is already in core.
4. An empty Context object may created at any time, quite easily, and there is already an established way to handle "no data here" for a context value. That means, like configuration, if we hit an edge case (such as maintenance mode) we can simply pass in an empty context object, so no harm done.

We therefore concluded that it was safe to make plugins depend on both context and configuration, as both systems should be available in any normal cirumstances and in exceptional circumstances (unit testing, maintenance mode, etc.) we can easily mock them to keep code working without having to scatter if (defined(MAINTENANCE_MODE)) calls all around the place.

Implementations

While every plugin implementation requires a class, it does not require a unique class. That is, two plugin implementations may use the same class with different configuration parameters to provide different functionality.

Taken to its logical conclusion, it should be possible for all plugins of a given type to use the same class and not even have to define it in each plugin implementation. Take, for example, layout plugins (as currently used in ctools/planels, but will also become necessary for core). A basic implementation of those could be (note, this is not even close to final syntax; just look at the concept):

plugin.type.core.layout.xml:

plugin.implementation.core.layout.mylayout.xml:

/core/includes/Drupal/Plugin/Core/Layout/Factory.php:

That is, any layout plugin that does not specify a class simply inherits the same common class, and that class knows exactly what to do with CSS files, template files, etc. Most people implementing a layout plugin would never see the word "class", unless they're documenting their CSS.

Similarly, subclassing off of another plugin's class and making only slight changes is also perfectly legal and appropriate. That should greatly simplify the implementation of, for instance, Field API Fields, Widgets, and Formatters.

Namespaces

How plugin classes are namespaced is dependent on how we end up structuring namespaces for Drupal in general, which is not yet decided pending performance investigation. However, for the time being we decided to use:

That is subject to change as core figures out what it's going to do.

Saving configuration

How one would go about writing to plugin configuration, say to change what subsystem was configured or to change block layout, we're still not sure. In a large part that still depends heavily on the outcome of the Configuration Management Initiative, so for the time being our answer is to punt on that until we know what that API will look like in more detail.

Edge cases

There are, of course, a lot of systems that don't quite fit this model directly. They may have only a single instance active at any given time; they may not have any separate instance configuration, etc. One of the benefits of this approach is that such degenerate cases can still be supported by the same system.

For example, we spoke with Dave Reid, one of the maintainers of the Media module, at DrupalCamp Austin. The Media module needs plugins for different media browsers (local files, YouTube files, etc.) However, there is really no reason he could see for them to have multiple instances, since Media would only ever iterate over them and throw them into separate panes in a dialog. In that case, the factory would silently create an instance in the background for each media browser plugin implementation, and then load and return it. Everything else about the API is exactly the same, with all of the consistency, improved DX, and "less work for Dave" benefits that come with that.

We also spoke with Commerce maintainer Ryan Szrama (who was also present), since Commerce is heavy user of the info hook pattern and a likely candidate to leverage core plugins heavily in Drupal 8. From a limited examination it appeared to work for his use cases as well, and the ability to easily subclass another plugin should help reduce code duplication. The switch from info hooks to configuration files did not seem problematic.

Conclusion

This is a long writeup, as we wanted to be thorough. (We probably still forgot something.) However, the underlying concepts are not actually that complicated. A Plugin Type is simply a PHP interface; a Plugin Implementation is simply a PHP class; a Plugin Instance is simply an Implementation plus configuration; a Plugin factory is just that, a standard factory object, common to a myriad of systems in a variety of languages.

While there are some potential DX pitfalls, most notably the separation of the config file and the implementing class, we believe that the unification that this approach provides us will be well worth it, and once more widely implemented will become the expected approach. "Config file + Class" is a common model in many PHP frameworks, including Symfony2, so it's not invented out of whole cloth.

The runtime cost should also be fairly cheap, assuming the Configuration API is not too expensive. (And if it is, we will have larger problems anyway.)

EclipseGc and neclimdul have already started putting together a prototype implementation in the WSCCI sandbox, in the wscci-plugins branch. Have a look in /core/includes/Drupal/Plugin if you want to see what the code could look like.

Thanks again to everyone who attended the sprint. Thank you also to Four Kitchens for letting us take over their conference room all day to hash this stuff out. It was one of the most productive meetings I've had in a long time. And thank you to Pantheon for buying everyone a proper Texas BBQ for lunch.