Phase 2: Response controllers

Note. Braindump below. Some thoughts might not make much sense and it's 11PM and I'm tired (teenagers need sleep more than anyone).

"1. Drupal builds the context object as above."
Isn't the whole point to lazy-load everything - meaning we *don't* build the context object?

This is where it gets sticky. We need to load as little information as possible. But if we follow the current block visibility pattern instead of a more page manager-esque pattern, if you have 50 blocks which each need a different part of the context you're going to end up having to load the entire context. Not good.

That's why the page manager solution is good - to recap: the page manager (at least currently) starts with a path and says "ok, there are five different pages we could potentially show at this path (the current "page" would be a given unique set of active handler + any blocks). I need foo, bar, and baz to be able to determine exactly which page to show". That way we only end up loading foo, bar, and baz instead of the whole context.

Number 3 looks extremely interesting. Does this pretty much mean that every page callback will become a class which has different methods such as ->returnHTML or ->returnJSON? That would be really, really awesome.

Potential problem: how do I use my module to add JSON callbacks to page callbacks provided by system.module. Simple answer: override the page callback and set it to a class which extends the system module. However, what if two modules want to do this?
Maybe we should go for the "fake objects" that are used in panels.

---

To address the discussion points:

Here lie the lines:
The Dispatcher will be swapped out once in a thousand years. It does need to be pluggable (install.php, update.php, authorize.php, etc), but almost no contrib should override it. Thus, it needs to be robust enough to not need to be overridden. It can't, for example, deal with RSS v. JSON. That needs to happen on some other level. If we go to above, maybe that's got to do something with the page manager-type-thing? This page returns JSON and this one RSS but they are all permutations of this other basic page. We determine, for example, based on HTTP headers whether it's JSON. That would need to happen at the level of the Response object.

Maybe the following happens (this ties into blocks TNG). Each block type object creates it's own response object (excellent for caching).

The Dispacher selects a set of display controllers. They fight it out to determine who gets to display. Each display controller can contain it's own set of response objects (blocks). The "main" page callback is essentially nothing - there's nothing "main" about it.

If the display controller needs JSON, it will ask it's response object(s) to return in a JSON format. If it needs RSS, same (typically an RSS display controller would contain only one response object).

---

Thus, if we go back to the page manager model, it seems like the Dispatcher is what's currently menu_execute_active_handler. The display controller is a "page" (not sure the specific page manager term). A response object is a block.

We probably need better names.

---

Phew, that was a lot. Hope that made sense. Probably didn't. I'm tired ;-).

[edit: fixed formatting]

First, +1 to Dries' note that these are really Response Controllers. At least, that's the outermost wrapper that we should be concerned with.

So I'm trying to think through this, and I think we're getting really muddy on what's the Response Controller, Display Controller, what the block does, etc... Basically, I think there's a very large number of permutations here, and some cases need tight coupling of an RC/DC/block concept, while it's completely irrelevant for others. Let me give some dummy examples to illustrate:

1. A standard, browser-based request to a site's homepage
2. A standard, browser-based request to node/%, on a site which has different display controllers for different node types
3. An ESI-ish request made by an intermediary caching agent for the HTML representation of a specific block on the node page from the last item
4. An AHAH request looking for HTML to replace into a form in order to reveal a dependent widget
5. An AJAX request looking for a JSON object with which to play
6. The same sort of AJAX request as in the last item, but with instructions in POST to package in some additional information
7. An request seeking RSS/ATOM output
8. A SOAP request that updates some server-side data
9. A 403
10. A batch API processing response
11. A form submission

This variety of requests encompasses the major axes of variation that I think we need to be cognizant of. Lemme dig in on what differentiates these:

1 & 2 are a well-understood Page Manager phenomenon right now; they'll probably only be differentiated in that the Dispatcher will be able to arrive at the DC & its conf using only $_GET['q] context data for 1, whereas 2 will require dipping into node context data. They are both generating a full HTML response, both using the standard Dispatcher, both need the theme system, and (let's assume) aren't dipping into the form cache. They both have a set of blocks that they need to arrange into a layout, and need to take care of attaching 'out-of-band' information. The role of the display controller here, if we're assuming it's something like Panels, is basically to facilitate the ordered placing and rendering of the blocks/components into their appropriate locations within the determined layout by interfacing the data objects with the theme system in an appropriate way.

3 is actually very similar to 1&2, but has some key differences: it's only generating an HTML fragment, and it needs to be able to set special HTTP headers for the caching agent. What makes it quirky is that there's a strong argument to be made for letting it cohabit paths with the whole page of which the requested block is a part - the reason for this being that it's easy enough to have a specialized, ESI DC that takes the loaded conf for the entire page, but then chooses to render only the element specified in the request.

4 is similar to 3 in that it is also an HTML fragment generator, but otherwise departs in some respects: it needs to interact with the form cache, it's only generating an HTML fragment, it's not assembling multiple 'blocks' together into a coherent whole (in fact, I think blocks are architecturally irrelevant to this case), and in 99% of cases, I don't think there's out-of-band information to attach. It's perfectly easy and reasonable to put these at their own request paths, though, which is another crucial difference from 3, and makes these much easier to handle.

5 starts getting us into clearly different territory, because it's the first of the non-HTML responses. The goal of the response controller here is no longer string formatting, but data object assembly, so the theme system is probably not needed. Fortunately, these requests probably don't need to cohabit with other different response types. But if these request types are still to utilize the same sort of block/component-based pattern that the other responses use, then the controller's job is to assemble data bits from each block and (let's say, for simplicity's sake) stick them into an associative array. Maybe we use a POST parameter to indicate the request as AJAX so we can do some early-phase switching, but that's why I included 6. 6 is generally very similar to 5, except that it presumably contains POST parameters that are irrelevant to routing in addition to the AJAX-indicating POST parameter. Not a big problem with a good implementation of context, I think, but the use of POST as both a container for data to be used by the application and a container for data used by routing logic gives it some architectural similarities to a form submission.

7, for me, sits instinctively between the JSON response and HTML responses; it's sorta data assembly, but still also sort of string formatting. I lean towards the former because we still don't need the theme system, as the output is primarily intended to be robot-readable. Also, similar to 3, we're in a situation where we're basically looking to output the content of what amounts to a single block. What makes this interesting is that RSS & ATOM are a particular form of XML - so it can't be as simple as calling ->returnXML(). IMO, this is a case where the role of the block is to either spit generic XML or a data object, then the RC transforms that output into the desired format. But there's relatively little use for a DC, here.

8 is cool because we can make a really early, definitive decision about the type of response we're processing based on the Content-Type header; this could even be a case where we switch out the standard Dispatcher. Assuming we stick with the normal Dispatcher, though, we're in a situation similar to 5&6, where breaking things up into a DC and blocks doesn't really make much sense, except possibly as a way of sticking data into an associative array to be rendered into XML. Also, this request is intended to accomplish a server-side write operation as well as generate a string response, which makes it a hybrid operation that a conventional read-focused DC isn't well-suited to. It also splits the system between potentially cacheable (read/response) segments and non-cacheable (write/process) segments. Bottom line - I think this is a case where it makes sense to have a robust RC that is itself, basically, a SOAP server.

9 is a bit of a trick because it should be a reminder that, in introducing all these different ways that Drupal CAN respond, there's an in-tandem requirement that our 403s vary their form based on the response controller that's handling the request. Some food for thought, there :)

10, as Larry earlier pointed out, would format its response as JSON, but is otherwise more analogous to what I blathered about wrt SOAP: a DC makes little sense here, because the role of the RC is just to trigger some server-side processing. Blocks are irrelevant, as is caching, and the only conf this thing wants is enough context to be able to reconstruct where it is in the current batched job to do the handoff.

11 is all POST processing and no rendering, ultimately terminating (maybe) in a redirection. DCs, blocks, caching, all that are irrelevant; all that matters is a thin RC that handles the form processing. An interesting thought on this - might we write a special Dispatcher for handling form processing? Surely we can make that process less complex than that required for a standard page request...

Whew, ok, my brain is starting to give out and I know I'm rambling, but let me try to sum these up into a list of major different axes that we ought to consider here:

1. What sort of string should be returned - HTML (nos. 1, 2, 3, 4, 9)? JSON (nos. 5, 6, 10)? XML (nos. 7, 8)? Nothing (no. 11)? And at what point, and by what mechanism, do we definitively determine this?
2. Whether or not this response should be able to 'cohabit' with another, very different sort of response controller at the same basic $_GET['q'] - definitely (1, 2, 3, 7, 8, 9), probably not (4, 5, 6, 10).
3. Is it at all sensible to utilize a Display Controller/blocks for this type of Response? Yes (1, 2), maybe (3, 5, 6, 7), no (4, 8, 9, 10, 11)
4. Is the response cacheable? Yes (1, 2, 3, 4, 7), maybe/partially (5, 6, 8, 9), caching is irrelevant (10, 11). And, how cross-applicable are different caching systems to these different types of response systems?

There a a couple things I'm hoping will emerge from this mess. First, and maybe most important, is that I'm coming to be of the opinion that we should have a Response Controller interface & families of classes, whatever, which is the thing that picks up when the Dispatcher is done. That Response Controller handles the scut work of doing things like setting HTTP headers and attaching out-of-band data, triggering processing that needs to occur, and only in cases where it is appropriate, use a 'Display Controller' pattern (or if we can manage it, interface) to assemble together a bunch of different content bits for display. But the DC pattern is optional, and at the discretion of the RC as to whether or not it's appropriate to use. All the DC really does, then, would be to take assemble blocks into an appropriate rendering order, tuck em into a layout, maybe apply some styling, integrate with caching, and follow the renderer's instructions on what kind of output they should request from the block.

And, related to that...I think this direction where we think of the blocks as themselves responders makes very little sense. Maybe there's a very thin responder that interacts directly with the blocks, or a thin responder with a thin DC that interacts with them, but that layer needs to stay simpler: take context, render some stuff. Blocks can, say, render out an HTML string, some kind of generic XML string, a pass-the-buck-on-rendering data object, or some other data format. But that's it. Somebody who just wants to knock out a a quick block that outputs the site slogan should not have to think about RSS. There's a scarier question lurking here, if you want it - if we do want to use the block pattern all over our Brave New Drupal, then can every type of DC use every type of block?

Honestly, having been writing this post for more than four hours now, my recommendation would be at this point that we make a serious effort to map out ALL the different types of requests/responses that we want to be able to make Drupal to handle, whether in core or via contrib. I think it will make further work on the Dispatcher, RC, etc., far easier, as it will give us a much better sense of what decisions to make where.