German WSCCI

It was a productive week in Munich! Of particular note, we have several new volunteers who have committed to working on various parts of the system as well as, finally, a clearer picture of how WSCCI and SCOTCH are going to communicate. (The people involved have always been communicating, of course; I mean where the code from WSCCI and the code from SCOTCH will interact.)

The battleplan

For the WSCCI Core Conversation, I identified a number of remaining tasks (some of which already had issues, and some that did not) that were important for the completion of WSCCI. As of right now, we've added a lot of plumbing to core from Symfony but have not yet really taken proper advantage of it. If we shipped today, we'd have a highly over-engineered system with few user-facing advantages over what we had in Drupal 7. Now comes the fun part. The complete list can be found in the slides from that session.

Afterward, several people stepped forward to help drive WSCCI home. We held a lengthy meeting where we mapped out who was going ot take point on which parts, and hopefully minimize the dependencies between them.

The main focus at the moment is the new routing system, which Larry is focused on at the moment. (More on that below.) reptilex offered to help with that, as the patch has been sorely lacking in reviews to date.

Now that we have this new REST-capable system, one of the first Drupal systems we need to improve as a result is our Ajax system. There is already an open issue with a description of what needs to be done. Kat Bailey and Kristiaan Van den Eynde offered to take point on that given their strong Javascript background.

Will Milton (wamilton) has been looking for backup on content negotiation for some time. Since this is an area that affects many PHP projects, not just Drupal, Lukas Smith has put out a call to other projects to collaborate on a solid PHP implementation. However, it looks like we won't be able to get that written by feature freeze. Instead, the reduced-scope goal is to build a simple interface for matching a list of allowed mime types against the mime type of a Route object, and then a very basic exact-string-match implementation behind it. Core can ship with that, but thanks to the interface we (should) be able to swap it out in contrib for something that bridges to whatever new library ends up getting collaboratively written (if any). Niels van Mourik (nielsvm) and Raymond Muilwijk (r.muilwijk) have offered to help wamilton with the basic implementation for core.

Of course, the new routing system won't really get us the benefits we need unless we are able to leverage HTTP caching properly. Mark Sonnabaum is already working on that, and Raymond Muilwijk (r.muilwijk) has offered to help with reviews and additional code.

Amongst the refactoring that we are now able to do is moving the path alias system to a proper self-contained library. Kat Bailey is already working on that, and Ronald Ashri (ronald_istos) will be joining her in driving that home.

In order to fully leverage HttpFoundation, we need to finally untangle our session handling system so we can replace it with the session handling built into the Request object, the code for which we are already carrying around in core. Pierre Rineau (pounard) has had a start on that for some time, but it's been stalled for a long time waiting on reviews. Andrei Mateescu (amateescu) has offered to help get some movement going on that issue.

Although not critical for web services per se, we want to leverage the Generator capability of Symfony's Routing component to separate paths from route names. (See the earlier post for background.) Peter Wolanin is still working on this issue in a separate branch.

Another feature that we will be able to support is earlier form processing. Although we will not be adopting Symfony's Forms component in Drupal 8, we will now be able to make some improvements to our own Form API. Specifically, we can process submitted forms earlier in a request listener, before we even bother routing the request. Alex Bronstein (effulgentsia) and Raymond Muilwijk (r.muilwijk) offered to work on that issue.

There is work upstream in Symfony to improve streamed file responses, which would allow us a nice performance improvement for private file handling and similar "file served through Drupal" cases. Niklas Fiekas has a very solid start here, but less time to work on it. Jeroen Van Beirendonck (oenie) has offered to back him up and help drive this home.

Finally, there's a nice big elephant in the room: Menu links. After we've ripped routing out of hook_menu, then what? What becomes of the menu links system, or things like page title? The general consensus seems to be that menu links should become entities. Then, the outline capability of the menu system should be factored out into a separate subsystem that is maintained as a denormalized table (or alternate non-SQL backend!), and is usable by menu links, book module, and potentially other systems. The Tree module was discussed as a potential source of inspiration and/or code. Once all of that is done, a bridge system can optionally be built that automates menu link creation from Routes without integrating into the Route system itself. What happens to page title callbacks and such, we didn't really figure out. A large number of people offered to help figure this question out, however, including: Peter Wolanin, Ronald Ashri (ronald_istos), Pedro Lozano, Andreas Hennings (donquixote), and Martino Piccinato. This is a critical question, as we cannot leave the menu links system orphaned.

Mixing WSCCI and SCOTCH

Far from being an idea your doctor would never approve, re-merging WSCCI and SCOTCH (the latter of which is a fork off of the former) is critical to the success of both initiatives as the primary goal of WSCCI is to enable SCOTCH. (I'm sure all the alcoholics in the audience are swooning about now.) I had a series of conversations throughout the week with SCOTCH lead Kris Vanderwater (EclipseGc), Sam Boyer (sdboyer), Alex Bronstein (effulgentsia), Peter Wolanin, Moshe Weitzman, and Symfony representatives Fabien Potencier and Lukas Smith about this all-important question. By the end of the week, we managed to reach what I believe is a solid way forward. To wit:

• One of the confusion points we've had is that Symfony controllers do not map exactly to either page callbacks or delivery callbacks in Drupal 7, so we need to shift our organization somewhat. Which are they closer to? What we've concluded is that they are something of a combination of delivery callbacks and "render plugins" from Panels (which is what supports things like the Panels In-Place-Editor). However, that's not an exact match.
• Therefore, all module-defined Routes will, by default, not specify a controller. Rather, they will specify a "body callback" of sorts, which is similar to a page callback today. (We may call this a subcontroller, inner controller, body controller, etc; not sure yet.) Then when processing a request, after we've found the appropriate route if the route does not have a _controller key specified then we will supply a default, which may even be mime-type-specific. That may be done via a request listener that happens after the Matcher fires.
• For now, that default controller will be a direct port of the HTML code in the ViewSubscriber class, which in turn is a direct port of the old HTML delivery callback, drupal_deliver_html_page().
• SCOTCH can then come in and replace that default controller with something more powerful and SCOTCH-y. And, worst-case scenario, that should even be possible in contrib if it doesn't manage to get into Drupal 8 core. (But please, let's not face that worst case scenario. Go help with SCOTCH!)
• When we compile our Routes, we will have to multiply them. That is, if a Route exists for /admin/config, then we will also create a Route for /_internal/block/admin/config (or something like that) which has the body callback AS its main callback. That allows the body area to be called as a subrequest for rendering the body, the exact same way as any other block. Which is exactly what we want, as then we get all the fun automatic HTTP caching we're looking for.
• There's probably a lot of optimiation around what should be a subrequest vs. an inline include vs. user configuration vs. etc. That will not be dealt with in the initial patch. Once that's in, we can deal with optimizing our subrequest handling. For now, we'll just make it work.
• Because of how the above is built, we still have the ability for a Route to declare a main controller itself and take over handling the entire request. Ajax requests, autocomplete, private files, etc. will likely do that, but if someone wants to implement their own Panels IPE-esque interface, or wants to just take over the entire page completely and ignore Drupal entirely, that becomes very easy to do. That opens up a lot of interesting possibilities for future contrib-based expansion.

Much of the above should happen as part of the Router patch, but it's possible that I'll ask to spin some of that off to a subsequent issue to keep things manageable. TBD as it gets implemented.

Call to arms

That's a lot. We've got three months. This will be a challenge. I think we're up to it, though.

Of course, there's also lots of other, smaller issues. For instance, hook_init(), hook_boot(), and hook_url_inbound_alter() can probably all now go away, as they have Symfony-event based alternatives that don't suffer from the same "is the code loaded yet?" chicken-and-egg problem that plagues Drupal's early bootstrap. (I'll be spinning up some issues for that as soon as it's not 1 am. :-) ) There are many systems in core that can and should get refactored into injectable objects that we place in the Injection Container, as that also helps to improve testing and streamline (and thefore make faster?) bootstrap. And of course there's plenty of other issues that still need to get tackled.

Interested but not sure where to start? Contact Crell or come by #drupal-wscci in IRC any time and we'll try to get you pointed in the right direction. We're going to need all the help we can get to drive this thing home.