About a month ago, the WSCCI team posted a core patch for the new Context API we've been working on. As predicted there was a fair bit of feedback, much of it resistant. A number of responses essentially felt that they could not tell if the new API was over-engineered or not as they had a hard time understanding the use case for much of the functionality that was built in, such as the "stacking" logic or nested context keys.
At the WSCCI team meeting on 6 December, we decided to take a step back and experiment a bit to see if we could demonstrate the need for that functionality with a quick'n'dirty simpler implementation. I volunteered to try to make such a demo using the Pimple dependency injection library (which was already reasonably similar in concept to the Context API as we had implemented it).
However, after some additional research and discussion with Symfony2 folks I realized that what we were trying to do... Symfony2's component libraries already do. After a long weekend of experimentation, I am actually ready to suggest taking a different tactic, somewhat as suggested (coincidentally after I did most of the experimentation) by Dries.
In short, I propose that we cut to the chase and port Drupal to Symfony2's HttpKernel library, and accept the rather drastic changes that will result.
Now before anyone starts sending the jedi ninja assassins after me give me a moment to explain.
The big picture
First, let's take a look back at what WSCCI's stated goals were. From our mission statement:
The Web Services and Context Core Initiative (WSCCI) aims to transform Drupal from a first-class CMS to a first-class REST server with a first-class CMS on top of it. To do that, we must give Drupal a unified, powerful context system that will support smarter, context- sensitive, easily cacheable block-centric layouts and non-page responses using a robust unified plugin mechanism.
- The future of the web is in REST and hypermedia communication. HTML pages are just a subset of that. Right now, Drupal core is pretty lame at that.
- Where pages are concerned, our current "inside out" push-based model is too limiting and inflexible. But it's baked very deeply into Drupal, and the only available solution today is Panels, which has to fight hard to get around Drupal's built-in assumptions. Plus, the future is not page-oriented, whereas Drupal 7 is even more locked to "whole page at once" logic than ever before.
- The changes necessary to make Drupal rock at both of those tasks (rather than simply tacking them on as an after-thought) are very deep and fundamental to the way Drupal works... but are in practice the same changes for both problems.
- So, WSCCI is about making those changes so that we can succeed at both of those tasks.
In the ideal, the flow of logic in a fully WSCCI-ified Drupal (as described back at DrupalCon San Francisco and fairly consistent since) would be:
- Request arrives
- Mutate request into a Context object, which contains (or has access to) Drupal-specific information
- The Context object is mapped to a controller based on both HTTP information (path, Accept header, etc.) and Drupal-specific information (e.g., node type)
- That controller returns a response
- If that controller is a page, then it has sub-components that themselves return a response. These subcomponents are called "blocks" (although they are more akin to Panels panes than core blocks). And you can override, manipulate, and mess with the Context object before passing it along to blocks.
- Recurse all the way down as far as you want
And because each component can lazy-load itself using classes, and we pass the request along to each controller, we get the following benefits as well:
- Smaller bootstrap through autoloading
- Asynchronous rendering of blocks, which means they can be cached independently for ESI, backbone.js, and other such hotness
- Injected data (from the request context that we pass in) means that we can more easily unit test parts of the system (DrupalUnitTestCase) rather than integration testing (DrupalWebTestCase)
That's the world that WSCCI has been attempting to work toward.
On Friday 9 December, I was in the #symfony-dev channel and ended up having a lengthy conversation with a very nice French gentleman named Christophe Coevoet (Stof). (I've attached a log excerpt to this post.) Stof stayed up well past his bedtime to help explain to me how the HttpKernel component of Symfony (which, like HttpFoundation, is easily spun off as its own library) works. After some discussion, he offered this off-the-cuff prototype of what Drupal might look like on HttpKernel (give or take some discussion and typos).
In short, the flow of control with HttpKernel is as follows:
- Request arrives
- Fire an "event" for "we're about to handle a request", which lets code modify and add to the request object.
- The request object is mapped to a controller based on both HTTP information (path, Accept header, etc.) and whatever other information you want to build into your mapper.
- That controller returns a response object
- Because all controllers are objects that take only injected information (really, all that makes an object a controller is that it takes a $request object and returns a $response object), it's dead-simple to nest them inside each other and simply combine their response objects as they get rendered. And you can duplicate, manipulate, and mess with the request object before passing it along to sub-controllers.
- Recurse all the way down as far as you want
In practice, the only substantive difference is that the request object carries around extra context rather than the context object carrying around a request. Which, in the grand scheme of things, is not a particularly significant difference.
Based on that conversation, I took a few days to try and turn that prototype into a working proof of concept. The result of that work can be seen in the wscci-symfony-dic branch in the WSCCI sandbox. Note: I have EventDispatcher and HttpKernel pulled in as git submodules from the latest development branch, so after checking out the repository you'll need to run
git submodule init && git submodule update to get a working system. (No, really. Everyone I've had preview this writeup has forgotten that part. The code won't work if you don't do that.)
It's still a rough proof of concept. A lot of things are hard coded and somewhat hacked, and there's probably a number of things that could be done more cleanly, and will be once we dig in further. However, I think it reasonably demonstrates the model that we want, end-to-end, as well as its advantages and the reason why we need certain functionality.
And, I don't think this can be over-stated, this is the result of a long weekend or so's worth of work. Most of the conceptual functionality we want is already here, in a form that is more well-tested than what we could do ourselves. While I have no doubt that the team working on WSCCI now could implement something along these lines from scratch, and we had intended to do so, I think we will get a better product by adopting code that already follows the model we wanted to follow, and has already thought through many of the challenges that we will run into if we try to do it ourselves. In IRC at the last WSCCI status meeting, in fact, almost everyone who's been involved with WSCCI to date was supportive of this move.
I encourage everyone to check it out and play with it a bit before commenting. I have left a number of comments in the code to help explain what's going on, and I have a more detailed writeup below.
Once you have the code checked out, start off normally with Drupal. Install it, create a node, etc. The index.php pipeline has not changed at all.
Once you have a node, go instead to
symfony.php?q=node/1. That renders with the new pipeline (mostly).
Looking at the code, start in symfony.php. All it does is setup the autoloader (mostly copied from bootstrap.php now), create a new request object, and pass that to the kernel. The kernel, by definition, returns a Response object, which is send()ed. The response object contains the body of the HTTP response, headers, methods to manipulate them, etc. So send() will send all appropriate headers, then the body, then we're done. (Actually right now I'm still calling drupal_page_footer(), but I suspect that could be turned into events that fire on the kernel.RESPONSE event. More on that in a moment.)
(Yeah, I had to get TNG in there somewhere.)
Now look at DrupalKernel. It implements the very thin HttpKernelInterface, which is designed to be stackable. This is important. Any HttpKernelInterface object takes a request, and returns a response. And because that's all it does, you can nest them inside each other.
In this case, we are running through a new Drupal bootstrap process. For now I have a call to the old one hacked in, but that will go away eventually once stuff gets moved over to the new code.
Next, we wire up a \Drupal\ServiceContainer. By "service" here we don't mean web services, but code services; things that you call to do stuff. This is a "Dependency Injection Container", which is simply the scariest word we could think of to describe an array of objects on steroids. (I actually very much encourage people to see that slideshow.) In this case, we're using Pimple, which is an extremely small DIC based on the same lazy-load principles as the Context API. (Context API actually took much of its concepts from Pimple.) Symfony2 the full framework uses its own DIC that I don't think will work for Drupal for various reasons. (Mostly, we cannot expect users to have shell access or run code generation every time they change a setting.)
The Drupal ServiceContainer is a Pimple subclass that sets up some core services by default (and we'll add more as time goes on), all of them in lazy-load form. And because they're lazy load, it really doesn't matter what order things get added in. Then, we let modules add their own services. To do that, we load up a specially named class from each file. That class is, in a sense, a class-based hook. However, using the same OO model as everything else allows us to very easily lazy load code, unit test those classes, and organize our code more consistently. It also means that, as noted in the comments, all we need is a list of module names and an autoloader that handles modules. (There's active discussion of that right now, and actually this model would recommend a module-centric rather than subsystem-centric namespace pattern.)
That means that at this point, all we've initialized is one small class per module (if it exists), and a few core classes. Maybe a DB connection if we need that for one of the module service registration classes, but maybe not. (That assumes that module_list() becomes based on CMI, not the database, but if not, no biggie.)
Next, we setup event handlers for the request. This is a built-in part of the default HttpKernel class, which we leverage. This works essentially the same way as the "manual" approach above for services. In this case, instead of wiring up code services we stick extra data onto the request object.
Normally, HttpKernel assumes you're using the $request->attributes property, which is there for exactly that purpose but, sadly,not well-documented as such. However, that is a front-loading container. What we want is lazy-loading of this extended data. Solution? Another instance of Pimple, which for historical reasons I'm calling $context. Note that we're using our own subclass of the Symfony request object, on which we are putting another Pimple instance. Then, any module can simply attach more lazy-loading context values in the exact same way (using the same code in the parent class, even) as services (and as the Context API). In this setup there is only a single key space; no nesting, no fallback, nothing.
Once those are setup, we create a resolver object. The resolver is simply the fancy name for "where what we currently call the menu system goes". But as an object, it's self-contained and encapsulated (and therefore testable, removable and replaceable, etc.). In our case we pass our service container to it so that it has access to any part of Drupal we want... but still no system has been initialized, and we could replace one with a stub for testing purposes easily if we wanted to.
Finally, we pass that resolver and the request to the HttpKernel class. It returns a response object, and we just return that response again.
Context, Take 2
Let's look at \Drupal\Module\node\ContextSubscriber. The EventDispatcher API is very simple. It lets you register methods on the object that will be called in response to certain events. Very hook-like, albeit with explicit registration. In onKernelRequest(), we set a context value for the current node, which is closure that will not execute until it gets called. It will also get called only once, thanks to Pimple. That means, for instance, the call to
$services'entity'->load(array($arg1)) doesn't happen until 'node' is first requested. And that means that the controller class is never even pulled off disk (assuming a PSR-0 autoloader) until then, nor is it instantiated. Nothing happens until we need it to.
Now let's go back to that dispatcher object. There are a number of ways that could get wired up, and HttpKernel has options I'm not even exploring yet for the sake of simplicity. Basically, what the dispatcher does is take the request and figure out which controller should handle it. The controller can be any callable; a function name, object-and-method, or a closure. I have standardized on an object-and-method, which I think is very wise as it's the only way we can lazy-load code.
For now \Drupal\Controller\DrupalControllerResolver has a few hard coded controllers and falls back to a port of menu_execute_active_handler(). Let's look at that first, further down.
Note that we are detecting the delivery callback first, and then passing the router item off to that controller. That lets the controller change what should happen, including handling of 403 and 404 errors, before we call the page callback, not after. LegacyControllerPage is essentially a port of drupal_deliver_html_page() to an object, which ends up cleaning up the code quite a bit (mostly by just breaking a big ugly switch statement into self-contained methods). That's how going to symfony.php?q=node/1 returns an actual page.
Now, let's look at a simpler "modern" case. In the dispatcher, we say that if there's a node in the path (meaning we're on node/1) and the request's Accept header is for application/json, we call a new controller, NodeJsonController. This controller is dead-simple; it grabs the node object out of $request->context['node'], renders it to JSON, and... doesn't return it. Instead, it creates a response object and sets the JSON string to the content of the response. The initializeResponse() method (which inherits from a parent class in this implementation) sets the appropriate headers, expiration time, etc. rather than having it in "some function somewhere", the way we do now. We could also set various and sundry other headers as appropriate, including an HTTP code other than 200. (Remember, this is a REST server. Use all of HTTP. If there's an error, then sending back a 200 response is broken.)
To test this, use a browser plugin of some kind to go to symfony.php?q=node/1 with an Accept header of "application/json". I recommend Poster for Firefox, but there are other options. You should get back a JSON string that maps 1:1 to a $node object. In practice we wouldn't just do json_encode(), but it works for demonstration purposes. Note here also that, were it not for the full bootstrap call we hacked in back in the kernel, nothing would have been loaded at all aside from the node object. If we don't use the form system or theme system or graph tracking library or file API, etc., none of it would have loaded, parsed, or even entered the code's imagination.
Now let's look at the other hard-coded controller, which responds to text/html requests for node/1/demo. This is how a "modern" page would be built. It would replace the inside-outside-step-twirl-dosie-doe process that is the current page rendering process, which has proven rather unpopular due to its impenetrable workflow.
Note that the controller we're loading up here is LayoutView. That is simply a special case of BlockController that has sub-blocks and puts them into a template. To quote Ross Perot, "it's just that simple". I didn't implement it here, but html.tpl.php and page.tpl.php would simply correspond to 2 different LayoutView instances. Let's actually look at BlockController first.
BlockController is an abstract class. It does the same thing as any other controller; it takes a Request and returns a Response. However, it is extended to include methods for CSS and JS tracking. As a block runs, code does not call drupal_add_css() but $this->addCss(). Why does that matter? Because in the execute() method we take those and attach them to the Response object! The response object is a Drupal-specific subclass of the HttpFoundation Response class that supports having CSS and JS tracked separately. As a result, every block is its own self-contained renderable thing. Any given block, we can get from it its CSS needs, its JS needs, and its response object (which contains its content). Normally, though, all we need externally is the response object.
Also see that BlockController renders any render array it receives on the spot. The response object cannot contain an array content, only a string. That's good, because it means we never have the epic array of doom that is the page array, which is horrid for memory as well as comprehensibility. We know, for certain, that each block is self-contained.
Let's look at \Drupal\Core\Controler\Block\NodeView. With most of the logic up in the parent class, all this needs to be is a port of the node_view() function. For now I have just inlined it. But, notice that the node object we're rendering is taken from the request context we were passed. We don't care at all where it came from.
To see why that matters, let's now look at LayoutView. For now it is completely hard-coded to this example, but in practice would be more generic and driven by configuration. In this case, we have 2 sub-blocks. For each sub-block, we do the same thing:
- Duplicate the request object.
- Modify it if desired.
- Pass the new request to the block and get the block's response object.
- Collect the response objects.
Once we've rendered all the child blocks, we aggregate their CSS and JS into our own response object, and then take the content of each request and throw it into a template file. (The details of doing that are rather hacky in this example, but I was going for simple and readable for the moment. This is just an architectural proof of concept, remember?) And then we get the output from that theme call and there's our "layout block" output. And because a layout is a block, we can nest them indefinitely.
Also, since Pimple treats explicitly assigned and lazy-derived values the same we can override any value we want. We can also explicitly put values into keys that a given block is expecting, just as Panels content panes do now; and we can even make those lazy-load, too.
Any page of any level of complexity can be built with that model, using those two simple concepts.
Whither the context stack?
The astute reader will notice something conspicuously missing from this model that was in Context API: The context stack and drupal_get_context().
First of all, the addLayer() method is effectively replaced with the existing Request::duplicate() method, which serves much the same purpose. It makes a new request/context object. In this case it clones the object, whereas Context API simply maintained a reference to the parent. Both are viable approaches; we originally went with the reference because it offered more flexibility with nested keys and caching, but if we don't do that then it's not necessary.
To explain the other, let's look at the current code and see what happens to $request. It is not passed any further than the controller by default. However, we could easily have code further down the line that needs it. How do we know anything down in hook_node_insert? Or an l() function? As this proof of concept is written now, you don't. That's a problem, however, for things that require, say, language. That is request/context information. How do we get that information as far down as a widget, or a field formatter, or an action, when the global $language variable is removed (which it will be, so help me Rasmus)?
There are three possible ways to deal with that:
- Say that code below the controller level just doesn't get to know about the request or context. It knows only what is passed to it directly, and if it needs more than that, well, the developer is SOL. I really don't think this is a viable option, especially for things like language that permeate Drupal so deeply.
- Allow a quasi-global version of the request object that such code can get at. Rather than a dozen random globals, global functions, and constants that flit about, we have one single global that we can easily get at and replace if necessary. That is, in short, drupal_get_context().
- Refactor all of those systems such that we can pass $request to them.
So far in WSCCI, we have had the position that #1 is not an option as it would be a massive feature regression and #3 was simply too ambitious. That left #2. Removing the request/context object from the stack after a block has finished, however, is critical. One mistake there could cause the entire system to go bonkers, and if it's done deliberately then it would break the caching we're able to do. (See below.) That is why Context API has the "forced pop" logic in it with the tracking variable.
For this model, so far, none of that is necessary. However, once we start moving more code into this approach we will be faced with the same question, and the same three options. Personally I still think that #2 is the safest approach, especially for hooks, but since we would, by necessity, be refactoring many other systems as well here it is not unreasonable to ask if #3 become something we could reach for. I am open to discussion on this point.
I clearly have not addressed access control at all in this scenario. Again, that's mostly for code simplicity. However, it would be quite simple to develop an access plugin interface and class, much like ctools access plugins today, that takes a $request and returns boolean true or false. Just like ctools access plugins we can then throw multiple of them onto a single block. We could conceivably run those separately from the controller itself (as part of a set of options stored in whatever the configuration store is for it) or put the configuration into the block and then call a method on it. There's pros and cons either way, but for now suffice to say that it opens up far more potential options than roles and a single menu access callback function today.
Caching and ESI
A final important point is caching. Right now I've not implemented any. However, HttpKernel includes an HttpCache mechanism that caches based on the $request object. And we could, conceivably, pull whatever we wanted out of it to use for caching purposes. See the last section in the original prototype for how relatively simple it is.
We can wrap an HttpCache of whatever kind we want (using whatever Store object we want) around any class that implements HttpKernelInterface. That's a nice and simple interface, so it would be trivial to throw onto every single block, simply by default. I believe all it would take is replacing the execute() method I added (purely a whim of a name at the time) with the interface's handle() method. "It's kernels all the way down!"
A very important side effect of that is we could then have the dispatcher map certain paths directly to certain blocks at any time, tweaking the request object as needed. That means... every single block can be its own request. That depends on blocks being asynchronous; which is why we need to never allow a block to have a direct effect on some other block on the page. If they're asynchronous, then every block (which includes layout of blocks) can be cached independently, rendered independently using ESI, requested from the browser, etc. And since the cache can be on anything from the request we want, we can cache blocks for authenticated users based on just the data that matters.
In that case, a "page" that lives in Varnish but is rendered by 3-4 calls back to the server that run in parallel each time, each of which is only a "partial bootstrap" (as we use the term now), becomes not just possible. It becomes fairly likely for many use cases.
In fact, HttpKernel ships with an ESI-aware reverse proxy cache implementation out of the box. Throwing that, or a subclass of it, onto every block by default (or on configuration) would not be difficult.
In all modesty, I don't know that we could pull something like that off on our own at this point. And if we did, it would probably be by just cloning this exact model. And if we're doing that, why not just use what already works?
See this introductory writeup for more.
Another advantage of the HttpKernelInterface is that, because it is nestable, you can nest any object that implements that. Even non-Drupal code. The full Symfony2 framework, Midgard, phpBB, Silex, and others use HttpKernel or will in their next version. That makes it possible to embed any of those applications in Drupal, or vice versa. The amount of sane "don't repeat yourself" that opens up is mind-boggling.
Just earlier this week, I was speaking with a developer who is working on, essentially, a stand-alone install-it-yourself clone of Disqus, using Symfony2. With HttpKernel and ESI support, that could run as a standalone app just like Disqus does or be embedded directly into any other HttpKernel-using application... including Drupal.
As I write this, Symfony 2.1 is nearing completion and I believe is expected in the January/February timeframe. However, 2.1 is not going to be an LTS release. That has been pushed off to a later version, as there are still some features that need to be worked out more. Some of the Symfony folks have specifically reached out to Drupal people to help fix some of those issues, because they are directly relevant to us. For instance, HttpFoundation's session handling needs love, particularly the session-messaging system (the rough equivalent of drupal_set_message()). This is a place that we can help direct Symfony development to our benefit, and the benefit of others, while benefiting from the combined work of all of those other projects that are also leveraging HttpKernel, HttpFoundation, etc. That is Open Source at its best, is it not?
I know this has been long, and I thank everyone who has made it this far. I wanted to make sure that I fully explained what is on the table, and why I believe it to be the right way forward.
In a sense, there's nothing really changing here. We still have the same end-game in mind as WSCCI has always had, even before it was called WSCCI. HttpKernel was on my list of things to look into eventually later, but it turns out, looking into it now is more productive. It also means that I really ought to file a documentation bug with HttpFoundation for not making it apparent what $attributes was for. :-)
I also freely acknowledge that those looking for a "modest" Drupal 8 cleanup-only cycle would be rather disappointed with this approach, as it is anything but modest. However, there is only so much cleanup that can be done without massive refactoring given Drupal's current architecture. As catch noted:
HTML5 is replacing our rotten windows (and fortunately replacing rather than re-painting since there is genuine refactoring of markup and CSS going on as well as introducing HTML5 stuff), WSCCI feels like building an extension and moving some cramped bits of the existing house into it. But there is also subsidence and termites in the basement.
So if the problem is a termite-ridden foundation, let's cut to the chase, build a new foundation that can withstand the world we live in now, and then move all of our nice antique furniture into it.
You may now release the ninja jedi assassins.