About a month ago, the WSCCI team posted a core patch for the new Context API we've been working on. As predicted there was a fair bit of feedback, much of it resistant. A number of responses essentially felt that they could not tell if the new API was over-engineered or not as they had a hard time understanding the use case for much of the functionality that was built in, such as the "stacking" logic or nested context keys.
At the WSCCI team meeting on 6 December, we decided to take a step back and experiment a bit to see if we could demonstrate the need for that functionality with a quick'n'dirty simpler implementation. I volunteered to try to make such a demo using the Pimple dependency injection library (which was already reasonably similar in concept to the Context API as we had implemented it).
However, after some additional research and discussion with Symfony2 folks I realized that what we were trying to do... Symfony2's component libraries already do. After a long weekend of experimentation, I am actually ready to suggest taking a different tactic, somewhat as suggested (coincidentally after I did most of the experimentation) by Dries.
In short, I propose that we cut to the chase and port Drupal to Symfony2's HttpKernel library, and accept the rather drastic changes that will result.
Now before anyone starts sending the jedi ninja assassins after me give me a moment to explain.
The big picture
First, let's take a look back at what WSCCI's stated goals were. From our mission statement:
The Web Services and Context Core Initiative (WSCCI) aims to transform Drupal from a first-class CMS to a first-class REST server with a first-class CMS on top of it. To do that, we must give Drupal a unified, powerful context system that will support smarter, context- sensitive, easily cacheable block-centric layouts and non-page responses using a robust unified plugin mechanism.
That is:
- The future of the web is in REST and hypermedia communication. HTML pages are just a subset of that. Right now, Drupal core is pretty lame at that.
- Where pages are concerned, our current "inside out" push-based model is too limiting and inflexible. But it's baked very deeply into Drupal, and the only available solution today is Panels, which has to fight hard to get around Drupal's built-in assumptions. Plus, the future is not page-oriented, whereas Drupal 7 is even more locked to "whole page at once" logic than ever before.
- The changes necessary to make Drupal rock at both of those tasks (rather than simply tacking them on as an after-thought) are very deep and fundamental to the way Drupal works... but are in practice the same changes for both problems.
- So, WSCCI is about making those changes so that we can succeed at both of those tasks.
In the ideal, the flow of logic in a fully WSCCI-ified Drupal (as described back at DrupalCon San Francisco and fairly consistent since) would be:
- Request arrives
- Mutate request into a Context object, which contains (or has access to) Drupal-specific information
- The Context object is mapped to a controller based on both HTTP information (path, Accept header, etc.) and Drupal-specific information (e.g., node type)
- That controller returns a response
- If that controller is a page, then it has sub-components that themselves return a response. These subcomponents are called "blocks" (although they are more akin to Panels panes than core blocks). And you can override, manipulate, and mess with the Context object before passing it along to blocks.
- Recurse all the way down as far as you want
And because each component can lazy-load itself using classes, and we pass the request along to each controller, we get the following benefits as well:
- Smaller bootstrap through autoloading
- Asynchronous rendering of blocks, which means they can be cached independently for ESI, backbone.js, and other such hotness
- Injected data (from the request context that we pass in) means that we can more easily unit test parts of the system (DrupalUnitTestCase) rather than integration testing (DrupalWebTestCase)
That's the world that WSCCI has been attempting to work toward.
Enter Symfony
On Friday 9 December, I was in the #symfony-dev channel and ended up having a lengthy conversation with a very nice French gentleman named Christophe Coevoet (Stof). (I've attached a log excerpt to this post.) Stof stayed up well past his bedtime to help explain to me how the HttpKernel component of Symfony (which, like HttpFoundation, is easily spun off as its own library) works. After some discussion, he offered this off-the-cuff prototype of what Drupal might look like on HttpKernel (give or take some discussion and typos).
In short, the flow of control with HttpKernel is as follows:
- Request arrives
- Fire an "event" for "we're about to handle a request", which lets code modify and add to the request object.
- The request object is mapped to a controller based on both HTTP information (path, Accept header, etc.) and whatever other information you want to build into your mapper.
- That controller returns a response object
- Because all controllers are objects that take only injected information (really, all that makes an object a controller is that it takes a $request object and returns a $response object), it's dead-simple to nest them inside each other and simply combine their response objects as they get rendered. And you can duplicate, manipulate, and mess with the request object before passing it along to sub-controllers.
- Recurse all the way down as far as you want
In practice, the only substantive difference is that the request object carries around extra context rather than the context object carrying around a request. Which, in the grand scheme of things, is not a particularly significant difference.
Weekend work
Based on that conversation, I took a few days to try and turn that prototype into a working proof of concept. The result of that work can be seen in the wscci-symfony-dic branch in the WSCCI sandbox. Note: I have EventDispatcher and HttpKernel pulled in as git submodules from the latest development branch, so after checking out the repository you'll need to run git submodule init && git submodule update to get a working system. (No, really. Everyone I've had preview this writeup has forgotten that part. The code won't work if you don't do that.)
It's still a rough proof of concept. A lot of things are hard coded and somewhat hacked, and there's probably a number of things that could be done more cleanly, and will be once we dig in further. However, I think it reasonably demonstrates the model that we want, end-to-end, as well as its advantages and the reason why we need certain functionality.
And, I don't think this can be over-stated, this is the result of a long weekend or so's worth of work. Most of the conceptual functionality we want is already here, in a form that is more well-tested than what we could do ourselves. While I have no doubt that the team working on WSCCI now could implement something along these lines from scratch, and we had intended to do so, I think we will get a better product by adopting code that already follows the model we wanted to follow, and has already thought through many of the challenges that we will run into if we try to do it ourselves. In IRC at the last WSCCI status meeting, in fact, almost everyone who's been involved with WSCCI to date was supportive of this move.
I encourage everyone to check it out and play with it a bit before commenting. I have left a number of comments in the code to help explain what's going on, and I have a more detailed writeup below.
Walkthrough
Once you have the code checked out, start off normally with Drupal. Install it, create a node, etc. The index.php pipeline has not changed at all.
Once you have a node, go instead to symfony.php?q=node/1. That renders with the new pipeline (mostly).
symfony.php
Looking at the code, start in symfony.php. All it does is setup the autoloader (mostly copied from bootstrap.php now), create a new request object, and pass that to the kernel. The kernel, by definition, returns a Response object, which is send()ed. The response object contains the body of the HTTP response, headers, methods to manipulate them, etc. So send() will send all appropriate headers, then the body, then we're done. (Actually right now I'm still calling drupal_page_footer(), but I suspect that could be turned into events that fire on the kernel.RESPONSE event. More on that in a moment.)
Bootstrap TNG
(Yeah, I had to get TNG in there somewhere.)
Now look at DrupalKernel. It implements the very thin HttpKernelInterface, which is designed to be stackable. This is important. Any HttpKernelInterface object takes a request, and returns a response. And because that's all it does, you can nest them inside each other.
In this case, we are running through a new Drupal bootstrap process. For now I have a call to the old one hacked in, but that will go away eventually once stuff gets moved over to the new code.
Next, we wire up a \Drupal\ServiceContainer. By "service" here we don't mean web services, but code services; things that you call to do stuff. This is a "Dependency Injection Container", which is simply the scariest word we could think of to describe an array of objects on steroids. (I actually very much encourage people to see that slideshow.) In this case, we're using Pimple, which is an extremely small DIC based on the same lazy-load principles as the Context API. (Context API actually took much of its concepts from Pimple.) Symfony2 the full framework uses its own DIC that I don't think will work for Drupal for various reasons. (Mostly, we cannot expect users to have shell access or run code generation every time they change a setting.)
The Drupal ServiceContainer is a Pimple subclass that sets up some core services by default (and we'll add more as time goes on), all of them in lazy-load form. And because they're lazy load, it really doesn't matter what order things get added in. Then, we let modules add their own services. To do that, we load up a specially named class from each file. That class is, in a sense, a class-based hook. However, using the same OO model as everything else allows us to very easily lazy load code, unit test those classes, and organize our code more consistently. It also means that, as noted in the comments, all we need is a list of module names and an autoloader that handles modules. (There's active discussion of that right now, and actually this model would recommend a module-centric rather than subsystem-centric namespace pattern.)
That means that at this point, all we've initialized is one small class per module (if it exists), and a few core classes. Maybe a DB connection if we need that for one of the module service registration classes, but maybe not. (That assumes that module_list() becomes based on CMI, not the database, but if not, no biggie.)
Next, we setup event handlers for the request. This is a built-in part of the default HttpKernel class, which we leverage. This works essentially the same way as the "manual" approach above for services. In this case, instead of wiring up code services we stick extra data onto the request object.
Normally, HttpKernel assumes you're using the $request->attributes property, which is there for exactly that purpose but, sadly,not well-documented as such. However, that is a front-loading container. What we want is lazy-loading of this extended data. Solution? Another instance of Pimple, which for historical reasons I'm calling $context. Note that we're using our own subclass of the Symfony request object, on which we are putting another Pimple instance. Then, any module can simply attach more lazy-loading context values in the exact same way (using the same code in the parent class, even) as services (and as the Context API). In this setup there is only a single key space; no nesting, no fallback, nothing.
Once those are setup, we create a resolver object. The resolver is simply the fancy name for "where what we currently call the menu system goes". But as an object, it's self-contained and encapsulated (and therefore testable, removable and replaceable, etc.). In our case we pass our service container to it so that it has access to any part of Drupal we want... but still no system has been initialized, and we could replace one with a stub for testing purposes easily if we wanted to.
Finally, we pass that resolver and the request to the HttpKernel class. It returns a response object, and we just return that response again.
Context, Take 2
Let's look at \Drupal\Module\node\ContextSubscriber. The EventDispatcher API is very simple. It lets you register methods on the object that will be called in response to certain events. Very hook-like, albeit with explicit registration. In onKernelRequest(), we set a context value for the current node, which is closure that will not execute until it gets called. It will also get called only once, thanks to Pimple. That means, for instance, the call to $services'entity'->load(array($arg1)) doesn't happen until 'node' is first requested. And that means that the controller class is never even pulled off disk (assuming a PSR-0 autoloader) until then, nor is it instantiated. Nothing happens until we need it to.
Dispatcher
Now let's go back to that dispatcher object. There are a number of ways that could get wired up, and HttpKernel has options I'm not even exploring yet for the sake of simplicity. Basically, what the dispatcher does is take the request and figure out which controller should handle it. The controller can be any callable; a function name, object-and-method, or a closure. I have standardized on an object-and-method, which I think is very wise as it's the only way we can lazy-load code.
For now \Drupal\Controller\DrupalControllerResolver has a few hard coded controllers and falls back to a port of menu_execute_active_handler(). Let's look at that first, further down.
Note that we are detecting the delivery callback first, and then passing the router item off to that controller. That lets the controller change what should happen, including handling of 403 and 404 errors, before we call the page callback, not after. LegacyControllerPage is essentially a port of drupal_deliver_html_page() to an object, which ends up cleaning up the code quite a bit (mostly by just breaking a big ugly switch statement into self-contained methods). That's how going to symfony.php?q=node/1 returns an actual page.
Modern controllers
Now, let's look at a simpler "modern" case. In the dispatcher, we say that if there's a node in the path (meaning we're on node/1) and the request's Accept header is for application/json, we call a new controller, NodeJsonController. This controller is dead-simple; it grabs the node object out of $request->context['node'], renders it to JSON, and... doesn't return it. Instead, it creates a response object and sets the JSON string to the content of the response. The initializeResponse() method (which inherits from a parent class in this implementation) sets the appropriate headers, expiration time, etc. rather than having it in "some function somewhere", the way we do now. We could also set various and sundry other headers as appropriate, including an HTTP code other than 200. (Remember, this is a REST server. Use all of HTTP. If there's an error, then sending back a 200 response is broken.)
To test this, use a browser plugin of some kind to go to symfony.php?q=node/1 with an Accept header of "application/json". I recommend Poster for Firefox, but there are other options. You should get back a JSON string that maps 1:1 to a $node object. In practice we wouldn't just do json_encode(), but it works for demonstration purposes. Note here also that, were it not for the full bootstrap call we hacked in back in the kernel, nothing would have been loaded at all aside from the node object. If we don't use the form system or theme system or graph tracking library or file API, etc., none of it would have loaded, parsed, or even entered the code's imagination.
Blocks
Now let's look at the other hard-coded controller, which responds to text/html requests for node/1/demo. This is how a "modern" page would be built. It would replace the inside-outside-step-twirl-dosie-doe process that is the current page rendering process, which has proven rather unpopular due to its impenetrable workflow.
Note that the controller we're loading up here is LayoutView. That is simply a special case of BlockController that has sub-blocks and puts them into a template. To quote Ross Perot, "it's just that simple". I didn't implement it here, but html.tpl.php and page.tpl.php would simply correspond to 2 different LayoutView instances. Let's actually look at BlockController first.
BlockController is an abstract class. It does the same thing as any other controller; it takes a Request and returns a Response. However, it is extended to include methods for CSS and JS tracking. As a block runs, code does not call drupal_add_css() but $this->addCss(). Why does that matter? Because in the execute() method we take those and attach them to the Response object! The response object is a Drupal-specific subclass of the HttpFoundation Response class that supports having CSS and JS tracked separately. As a result, every block is its own self-contained renderable thing. Any given block, we can get from it its CSS needs, its JS needs, and its response object (which contains its content). Normally, though, all we need externally is the response object.
Also see that BlockController renders any render array it receives on the spot. The response object cannot contain an array content, only a string. That's good, because it means we never have the epic array of doom that is the page array, which is horrid for memory as well as comprehensibility. We know, for certain, that each block is self-contained.
Let's look at \Drupal\Core\Controler\Block\NodeView. With most of the logic up in the parent class, all this needs to be is a port of the node_view() function. For now I have just inlined it. But, notice that the node object we're rendering is taken from the request context we were passed. We don't care at all where it came from.
To see why that matters, let's now look at LayoutView. For now it is completely hard-coded to this example, but in practice would be more generic and driven by configuration. In this case, we have 2 sub-blocks. For each sub-block, we do the same thing:
- Duplicate the request object.
- Modify it if desired.
- Pass the new request to the block and get the block's response object.
- Collect the response objects.
Once we've rendered all the child blocks, we aggregate their CSS and JS into our own response object, and then take the content of each request and throw it into a template file. (The details of doing that are rather hacky in this example, but I was going for simple and readable for the moment. This is just an architectural proof of concept, remember?) And then we get the output from that theme call and there's our "layout block" output. And because a layout is a block, we can nest them indefinitely.
Also, since Pimple treats explicitly assigned and lazy-derived values the same we can override any value we want. We can also explicitly put values into keys that a given block is expecting, just as Panels content panes do now; and we can even make those lazy-load, too.
Any page of any level of complexity can be built with that model, using those two simple concepts.
Whither the context stack?
The astute reader will notice something conspicuously missing from this model that was in Context API: The context stack and drupal_get_context().
First of all, the addLayer() method is effectively replaced with the existing Request::duplicate() method, which serves much the same purpose. It makes a new request/context object. In this case it clones the object, whereas Context API simply maintained a reference to the parent. Both are viable approaches; we originally went with the reference because it offered more flexibility with nested keys and caching, but if we don't do that then it's not necessary.
To explain the other, let's look at the current code and see what happens to $request. It is not passed any further than the controller by default. However, we could easily have code further down the line that needs it. How do we know anything down in hook_node_insert? Or an l() function? As this proof of concept is written now, you don't. That's a problem, however, for things that require, say, language. That is request/context information. How do we get that information as far down as a widget, or a field formatter, or an action, when the global $language variable is removed (which it will be, so help me Rasmus)?
There are three possible ways to deal with that:
- Say that code below the controller level just doesn't get to know about the request or context. It knows only what is passed to it directly, and if it needs more than that, well, the developer is SOL. I really don't think this is a viable option, especially for things like language that permeate Drupal so deeply.
- Allow a quasi-global version of the request object that such code can get at. Rather than a dozen random globals, global functions, and constants that flit about, we have one single global that we can easily get at and replace if necessary. That is, in short, drupal_get_context().
- Refactor all of those systems such that we can pass $request to them.
So far in WSCCI, we have had the position that #1 is not an option as it would be a massive feature regression and #3 was simply too ambitious. That left #2. Removing the request/context object from the stack after a block has finished, however, is critical. One mistake there could cause the entire system to go bonkers, and if it's done deliberately then it would break the caching we're able to do. (See below.) That is why Context API has the "forced pop" logic in it with the tracking variable.
For this model, so far, none of that is necessary. However, once we start moving more code into this approach we will be faced with the same question, and the same three options. Personally I still think that #2 is the safest approach, especially for hooks, but since we would, by necessity, be refactoring many other systems as well here it is not unreasonable to ask if #3 become something we could reach for. I am open to discussion on this point.
Access control
I clearly have not addressed access control at all in this scenario. Again, that's mostly for code simplicity. However, it would be quite simple to develop an access plugin interface and class, much like ctools access plugins today, that takes a $request and returns boolean true or false. Just like ctools access plugins we can then throw multiple of them onto a single block. We could conceivably run those separately from the controller itself (as part of a set of options stored in whatever the configuration store is for it) or put the configuration into the block and then call a method on it. There's pros and cons either way, but for now suffice to say that it opens up far more potential options than roles and a single menu access callback function today.
Caching and ESI
A final important point is caching. Right now I've not implemented any. However, HttpKernel includes an HttpCache mechanism that caches based on the $request object. And we could, conceivably, pull whatever we wanted out of it to use for caching purposes. See the last section in the original prototype for how relatively simple it is.
We can wrap an HttpCache of whatever kind we want (using whatever Store object we want) around any class that implements HttpKernelInterface. That's a nice and simple interface, so it would be trivial to throw onto every single block, simply by default. I believe all it would take is replacing the execute() method I added (purely a whim of a name at the time) with the interface's handle() method. "It's kernels all the way down!"
A very important side effect of that is we could then have the dispatcher map certain paths directly to certain blocks at any time, tweaking the request object as needed. That means... every single block can be its own request. That depends on blocks being asynchronous; which is why we need to never allow a block to have a direct effect on some other block on the page. If they're asynchronous, then every block (which includes layout of blocks) can be cached independently, rendered independently using ESI, requested from the browser, etc. And since the cache can be on anything from the request we want, we can cache blocks for authenticated users based on just the data that matters.
In that case, a "page" that lives in Varnish but is rendered by 3-4 calls back to the server that run in parallel each time, each of which is only a "partial bootstrap" (as we use the term now), becomes not just possible. It becomes fairly likely for many use cases.
In fact, HttpKernel ships with an ESI-aware reverse proxy cache implementation out of the box. Throwing that, or a subclass of it, onto every block by default (or on configuration) would not be difficult.
In all modesty, I don't know that we could pull something like that off on our own at this point. And if we did, it would probably be by just cloning this exact model. And if we're doing that, why not just use what already works?
See this introductory writeup for more.
External compatibility
Another advantage of the HttpKernelInterface is that, because it is nestable, you can nest any object that implements that. Even non-Drupal code. The full Symfony2 framework, Midgard, phpBB, Silex, and others use HttpKernel or will in their next version. That makes it possible to embed any of those applications in Drupal, or vice versa. The amount of sane "don't repeat yourself" that opens up is mind-boggling.
Just earlier this week, I was speaking with a developer who is working on, essentially, a stand-alone install-it-yourself clone of Disqus, using Symfony2. With HttpKernel and ESI support, that could run as a standalone app just like Disqus does or be embedded directly into any other HttpKernel-using application... including Drupal.
Working upstream
As I write this, Symfony 2.1 is nearing completion and I believe is expected in the January/February timeframe. However, 2.1 is not going to be an LTS release. That has been pushed off to a later version, as there are still some features that need to be worked out more. Some of the Symfony folks have specifically reached out to Drupal people to help fix some of those issues, because they are directly relevant to us. For instance, HttpFoundation's session handling needs love, particularly the session-messaging system (the rough equivalent of drupal_set_message()). This is a place that we can help direct Symfony development to our benefit, and the benefit of others, while benefiting from the combined work of all of those other projects that are also leveraging HttpKernel, HttpFoundation, etc. That is Open Source at its best, is it not?
Conclusion
I know this has been long, and I thank everyone who has made it this far. I wanted to make sure that I fully explained what is on the table, and why I believe it to be the right way forward.
In a sense, there's nothing really changing here. We still have the same end-game in mind as WSCCI has always had, even before it was called WSCCI. HttpKernel was on my list of things to look into eventually later, but it turns out, looking into it now is more productive. It also means that I really ought to file a documentation bug with HttpFoundation for not making it apparent what $attributes was for. :-)
What has changed is a less phased, more "all at once" approach. However, that seems to be what a number of people are asking for. (You asked for it, Dries...)
I also freely acknowledge that those looking for a "modest" Drupal 8 cleanup-only cycle would be rather disappointed with this approach, as it is anything but modest. However, there is only so much cleanup that can be done without massive refactoring given Drupal's current architecture. As catch noted:
HTML5 is replacing our rotten windows (and fortunately replacing rather than re-painting since there is genuine refactoring of markup and CSS going on as well as introducing HTML5 stuff), WSCCI feels like building an extension and moving some cramped bits of the existing house into it. But there is also subsidence and termites in the basement.
So if the problem is a termite-ridden foundation, let's cut to the chase, build a new foundation that can withstand the world we live in now, and then move all of our nice antique furniture into it.
You may now release the ninja jedi assassins.
| Attachment | Size |
|---|---|
| symfony-log.txt | 28.43 KB |
Comments
This is not Drupal
This will be a new project that adopts the Drupal name if we so choose. I have no problems with doing that, not anymore. Just let's be clear about what it is. Arguably, of course, Drupal 7 is a lot different from Drupal 4.6 too and it is possible what's necessary is one big jump instead of the four major releases it took from 4.6 to get where we are now.Edit: I wish I could delete all my comments but that I won't do. However, consider them deleted. Do not quote them or credit me with these opinions. They have been obsoleted.
and this is not a good direction either
Anything PHP, much less Drupal can not be a first class REST server. Leave Drupal the already awesome HTML generation project it is, make it more awesome doing that, fix the field UI, make media handling better and so on. As for REST? http://drupal.org/project/redis_ssi and http://drupal.org/project/varnish . (See http://drupal4hu.com/node/313 for a bit more. )Edit: left a note in the appropriate place aka. the core queue.Edit: I wish I could delete all my comments but that I won't do. However, consider them deleted. Do not quote them or credit me with these opinions. They have been obsoleted.
I will have a lot of replies
If we are married to the idea of a full hook replacement mechanism then why are we looking at Symfony? Yes, we picked Symfony HTTPFoundation but Drupal is AOP and so are Lithium filters and FLOW3. Why are we not shopping there for a new kernel if we want one? (Not that we want one, mind you.)Edit: I wish I could delete all my comments but that I won't do. However, consider them deleted. Do not quote them or credit me with these opinions. They have been obsoleted.
Indeed Symfony2 core does not
Indeed Symfony2 core does not make much use of AOP. That being said the standard edition of Symfony2 does ship with an AOP solution for adding security checks to controller methods:
https://github.com/schmittjoh/JMSSecurityExtraBundle
and there is also a Bundle for a more generic AOP solution:
https://github.com/schmittjoh/JMSAopBundle/blob/master/Resources/doc/ind...
This Bundle however seems fairly tied to the Symfony2 DIC.
It uses a so called compiler pass to scan through all services and their class definition to generate proxies and point the services back to those proxies. This in turn doesn't sound very compatible with the concept of Pimple and the use of closures, nor with the self imposed limitation of not wanting use code generation (which FYI FLOW3 is heavily based on). Here is the code for the compiler pass:
https://github.com/schmittjoh/JMSAopBundle/blob/master/DependencyInjecti...
Then again I could see going through all the services in Pimple, making instances of the services to determine the classes, then doing reflection to find any AOP annotations and then if needed wrapping them the Pimple service inside another one that implements what ever AOP magic that was annotated. But for that to be feasible you would once again need caching.
As for Lithium, I do not have much experience with it, but it draws heavily on closures, which imho should be used sparingly because otherwise it will become hard to manage very complex software. IMHO Lithium's closure derived flexibility mainly shines in highly custom applications that do not intend to really do a lot of code reuse between applications. But here I am mostly speculating due to lack of practical experience.
Which to me means its likely more feasible for Drupal to define a set of events that cover 80% of the use cases where AOP could also be used and letting developers add custom events along with overriding standard services to achieve the other 20%.
Any other approach imho will likely need code generation in order to be efficient. Now my understanding is that the Drupal security team does not like the idea of the web server user generating code that is then included. Here I see a few possibilities which all share the idea that if the generated code isn't available the fallback would be to generate it on the fly:
1) letting the users do the code generation "offline" and then uploading it via ftp
2) users that have access to the CLI would run an "optimize command"
3) running said "optimize command" via a cron job if the user has the ability to define cron jobs
I should note .. I am brainstorming here ..
Just checked Lithium (slides from dev team and core docs)
Lithium uses a system called "Filters", used to replace a central PubSub system, to enable an event-driven mechanism. The goal is to limit coupling and avoid the whole application being dependent of a central system. And Lithium doesn't have a Dependency Injection Container, because it's AOP, meaning you pass the configuration of a class as a simple array, that you can modify at runtime. Highly flexible.
The problem I see is that it doesn't help us because :
Lithium also has Request and Response objects, with a Dispatcher that uses the Router to find the proper Controller for generating the response. The approach looks similar to Symfony's HttpFoundation + HttpKernel + EventDispatcher + Controller, but as they don't by nature make use of DIC but config arrays only, I don't see them as good fit...
This is just based on a little observation of Lithium, I never used it. Any correction really appreciated.
Lithium Architecture and AOP
Hi all, thanks for taking an interest in Lithium. I'm Nate, the lead developer, and I'd be happy to answer any questions you might have, here or in private.
I understand your concern about layering multiple frameworks, but Lithium is fairly decoupled, and you'd really only depend on a few classes. We were actually already considering a separate release of just the infrastructure classes (none of the HTTP, routing, controllers, etc.), which I think would assist you if you were to use the filtering system.
By way of a quick primer, here's an example of how filters in Lithium work, using our request dispatching system, and (simplistically) caching by request URL:
<?php
// Static method:
Dispatcher::applyFilter('run', function($self, $params, $chain) {
// Check the cache for results by URL
if ($result = Cache::read("default", $params['request']->url)) {
return $result;
}
// No cached result, execute the request
$result = $chain->next($self, $params, $chain);
// Store the request in cache and return...
Cache::read("default", $params['request']->url)
return $result;
});
// Instance method:
$postsController->applyFilter('redirect', function($self, $params, $chain) {
// Clean up before redirecting...
return $chain->next($self, $params, $chain);
});
// Lazy-apply a filter to a static method, to delay loading of MyClass:
Filters::apply('MyClass', 'method', function($self, $params, $chain) {
// Do something...
return $chain->next($self, $params, $chain);
});
?>
By default, Lithium aggregates filters per-class or per-object internally. However, you could easily change this to a centralized dispatch system, even by extending the Filters class. All each class/instance really does is pass its list of filters to the Filters class for execution anyway, we just make a pretty API around it so the user doesn't have to do it manually on every method call.
We've had great success with the filtering system in Lithium, though it requires a bit of design discipline to use well. :-) Best of luck in your quest to make Drupal 8 architecturally badass. Let me know if/how I can help.
Thank you Nate !
I'm very pleased to see you, Lithium lead developer, taking the time to personally show some stuff from your framework. I think it shows how opened the PHP world is, and we (Drupal community) should take the chance to collaborate/learn from others as well.
Just a note : we (may) have a need to build a better PubSub system than our hook system, which is extremely simple to use but memory consuming. Feel free to correct me : the Static and Instance methods automatically load the corresponding classes (Dispatcher and classof($postController)) before even applying the filter and running it when necessary, whereas the lazy method waits until MyClass::method is actually called to run the filter, without loading itself MyClass, right ? Sounds very interesting for our case...
I already have a few questions :)
1) Can filters apply non-anonymous functions ?
2) Is it possible to have a listing of all filters that apply to a function ?
3) If 2), how efficient is the building of this listing ?
4) For the lazy method, can we apply a filter to namespaced classes and fully namespaced classes ? Like
Filters::apply('My\Namespaced\Class', 'method', ...)andFilters::apply('\My\Fully\Namespaced\Class', 'method', ...)? I don't know when and how the class names are resolved in Lithium, that's why I'm asking.Thanks in advance !
Any time
And thanks for the warm welcome.
Regarding class loading, you're absolutely correct. I noticed from the thread that memory efficiency is a major concern, which is why I dropped that last example in there with
Filters::apply().To answer your questions:
1) Currently... sort of. Though we don't use this, a filter could theoretically be a string reference to a function name. To use a class or instance method, you could do one of the following:
<?php
// Class implements PHP's magic __invoke() interface to make callables act like closures
class Invokable {
protected $_classOrInstance;
protected $_method;
public function __construct($classOrInstance, $method) {
$this->_classOrInstance = $classOrInstance;
$this->_method = $method;
}
public function __invoke($self, $params, $chain) {
if (is_string($class = $this->_classOrInstance)) {
return $class::$method($self, $params, $chain);
}
return $this->_classOrInstance->{$method}($self, $params, $chain);
}
}
// Then...
$custom = new \\my\\custom\\Class(); // This class has the method to be used as the filter
$instance->applyFilter(new Invokable($custom, "method"));
// Alternatively:
$instance->applyFilter(new Invokable("my\custom\Class", "method"));
?>
Or instead of a new class, you can just import an object instance into a closure when you attach it, and call the instance from inside the closure, like so:
<?php$hookObject = new HookObject();
$instance->applyFilter("method", function($self, $params, $chain) use ($hookObject) {
// Or use (&$hookObject)
$hookObject->something();
return $chain->next($self, $params, $chain);
});
?>
The other possibility is to modify the core to use
call_user_func_array(), but this would need to be tested thoroughly, as I'm wary of the performance implications...2) Sure. A bit of background: currently,
applyFilter()just stores each closure in an array inside the class or object instance. When a filtered method gets run, it passes the array of closures, as well as the method implementation itself (wrapped in a closure) to theFiltersclass, where everything gets processed. I'd imagine you'd want to implement your own filter management API to interact with theFiltersclass. But the short answer is yes, filters are just stored as an array. You could even index them by the class/module that's applying them.3) The listing is just built through calls to
applyFilter(), and it just ends up as a flat array. The great thing about closures/anonymous functions is that they're relatively cheap and they make everything lazy.4) When you're dealing with class names inside strings, they always have to be fully namespaced, but don't need leading backslashes. Again though, you could easily come up with your own mapping here that acts as an intermediary layer between the module API and the
Filtersclass. By default, Lithium works with any PSR-0 compatible class library out of the box, though we do impose some additional constraints on ourselves, which are documented here: http://lithify.me/docs/lithium/core/Libraries -- Naturally, non-conforming libraries can be loaded as well, there's just a bit more configuration involved.Interesting !
Thanks for your detailed answers, Nate. The fact that with this method we can make it all lazy loading gives me some hope...
@Nate and @Crell : I can't make up my mind, but something tells me that Lithium's Filter could help us replace hooks efficiently. Imagine something like this :
1/ Drupal is bootstraping
2/ At some point, core invokes a hook, say
hook_init. With Lithium's Filters, let's say we adopt the instance method on Controllers or Dispatcher (not sure...). So we call something like$instance->apply('CoreClass', 'hook_init, function($self, $params, $chain) { ... });3/ Before 2/, we build a list of all classes that implements this hook_init, using the method described in 2) by Nate above.
4/ Then we loop through this list and call each hook_init_implementations, using a LOC like the one in 2/ (2 items above).
5/ In fact, the LOC in 2/ shouldn't be using anonymous functions, but named functions using the 1st method in 1) by Nate above.
6/ Bonus : we can even somehow implementing module weight here.
I realize I'm not very clear... Let me try to rephrase it. This method replaces the loading of all modules, that then allows us to use the
function_exists()inmodule_invoke_all. Problem : this method needs 2 things : a useless call to the database, whereas we're just dealing with code, and loading all modules in memory, regardless of their utility in dealing with the request. What I'm trying to imagine is using Lithium Filter to lazy load what we now call "hook implementations".So if this method is possible, we need to solve non core module namespacing, as well as make sure that we can collect a list of modules implementing a hook before actually calling them. If it's possible, and with good performance, I think it's worth trying. Crell, Nate, others, is it even possible ?
Elsewhere?
While I appreciate the interest in improving the performance of hooks, I think that's off topic for this thread. I'd rather focus on the page building and bootstrap workflow, as that's what WSCCI is dealing with.
If you wanted to take the hook discussion to another thread, though, please reply here with a link so that I can follow it over there. I just want to keep this thread focused on the critical path workflow.
(I have my own ideas for how to deal with hooks that I've been trying to hold off on for now, to avoid gutting too much at once.)
Also note that the
Also note that the performance of hooks is a relatively small issue compared to other behaviours of the hook system (like inconsistent results from the same module_invoke_all() call depending on bootstrap level), or to performance issues with other subsystems in core.
I miss some developer
I miss some developer examples in this writeup. This makes it hard to discuss this proposal for 99% of the drupal users. It would be nice if you could map the examples we have in the issue queue to the new proposal.
EDIT:
If you are confused about the code (as I was) read http://www.slideshare.net/weaverryan/a-php-christmas-miracle-3-framework...
Everything makes more sense when you're done reading.
Indeed, a great slideshow!
Indeed, a great slideshow!
.
Thank you for writing this all out so carefully, especially the pointers regarding what code to look at. I think it's going to take some time to grok.
I also appreciate chx's point about scale. I don't understand enough to assess the comparison that this is equivalent to a jump from D4.6 to D7, but: during the fireside chat, Dries said something like "If we have a cleanup-focused cycle, maybe that cycle should be Drupal 9." (Paraphrasing because the recording is 1.5h long and I can't find it panning around for 15 minutes.) If we consider the direction outlined above, we should probably also consider something like that for the D9 cycle.
Edit: Also, as aspilicious says above, examples of how this would affect code that we write now would be very helpful. I too am most concerned with the potential impact on DX, because frankly I don't understand any of it enough. "This is what we write now" vs. "This is what we write after WSCCI" is what I think I and many others will need to react.
Thanks
I don't have nearly the knowledge necessary to evaluate the architectural suggestions or the implementations, but I do want to say that I'm very happy for this work being done. Thanks to all you people spending countless hours on pushing Drupal forward and exploring new possibilities.
Build from scratch
Modestly I think it will be much easier and faster to build Drupal from scratch in top of Symfony Standard Edition (https://github.com/symfony/symfony-standard), than refactor/adapt Drupal's current architecture to the approach described.
I don't see that really as a
I don't see that really as a feasible approach. However I do recommend that as many Drupal devs as possible try and play with Silex (probably easier to get to the nuts and bolts) or the Symfony2 Standard Edition. But do note that while what Crell proposes here will result in Drupal adopting a similar flow through the request, there is still a lot of freedom to define how controllers are discovered ("resolved"), what they bring with them out of the box etc. So really when I suggest to play with Silex/Symfony2 SE I am suggesting to look at the core flow through the respective Kernel implementations.
I think this would be a valuable experience that can help understanding the architecture that is being proposed here. Also please feel invited to join #symfony-dev on freenode IRC to discuss any questions that might come up in the process.
I don't think you can have a
I don't think you can have a "clean up" and yet run as many initiatives around improvements as 8 is doing. That simply won't work, so choose the best implementation for the job. If Symfony has already done this work, its preferable to take it on board than try and recreate it.
At face value, I understand this proposal better than the current implementation, and I contributed a very modest patch to that. I suspect this is because Request First, Context Second makes more sense than mixing the two, which was a bit complicated. Providing context as a quasi-global is a very simple way to deal with the resulting lack of a context object in the request. So, conceptually (and for DX) this change doesn't seem all that different to the changes brought by the original Context proposal anyway.
Id be interested to know how these changes might affect the plugin initiative, however?
The Cathedral and the Bazaar
We have a very good culture of reusing instead of reinventing, in our community. In everything from how modules and subsystems are designed, to how projects are invented, matured in contrib and later moved into core. This has been done (mostly) inside our own community, and worked great.
But as our project and our community grows, we have to find ways to scale how we've been doing things so successfully. Because I think it's not about what code we write, but how we do it.
I think reaching outside our community, to scale the way we reuse things, is the right thing to do. Again, it might not be what we used to do. But I think it's definitely how we used to do things. Just in a different scale.
To quote The Cathedral and the Bazaar, which I think maps well to our project:
So let's keep building Drupal the bazaar-way, as I think we have been doing so successfully. Simply put, I think what Crell is suggesting is really great.
Edit: Sorry for being political instead of talking implementation details. I'll have a look at those too ;)
Request with context = response with wrapper
I like this approach, if I understand it correctly a it retrieves raw data plus contextual components via context-appropriate vessel/template so that we don't have to confuse the model with the view.
Notice the IRC-log
Drupal 8 is for disruptive, vista-expanding changes like this
As someone invested in Drupal and not helping enough, i like the approach, interaction with Symfony2, and timing. If it means we can more easily Drupalize parts of Drupal that have been repeatedly passed over -- like making the one-way-only messaging system hookable -- that would be fantastic. The deciding factors would seem to be the impact on the configuration initiative and other core initiatives like the rationalizing overhaul of the theme system (homeless but maybe contained in the theming and clean up core initiatives).
benjamin, agaric
I tried to look into this,
I tried to look into this, but I'm having a very hard time to find the respective code you're referring to in this write-up.
E.g., where is the "Dispatcher"? There's no file with that name, and I'm not really able to locate the code you're referring to in the files. Thus, I can't really make sense of two entire chapters of this write-up.
While I may understand the excitement about contexts, classes, autoloading, plugins, performance, and whatnot, the repetitive notes about how that's awesome are frankly a bit disturbing while trying to grok this. And I have a very very very hard time to grok this, as you're essentially replacing the entire procedural Drupal architecture with object-oriented code.
That said, I highly appreciate that this write-up is finally one that's more concrete and attempts to provide a view on the big picture backed by actual code.
I'm not sure how others think about it, but for me, an even more straightforward, reduced to the max, super-concrete write-up and walk-through would be helpful. Many things might make more sense to explain in code comments even.
Thanks,
sun
Daniel F. Kudwien
netzstrategen
You probably didn't
You probably didn't initialize and update the submodules (EventDispatcher and HttpKernel are two git submodules).
(Still going through the code myself, a comment will follow)
Agreed
I'd suggest some simple, clear instructions, like:
git clone --branch wscci-symfony-dic you@git.drupal.org:sandbox/Crell/1260830.git wsccicd wsccigit submodule init && git submodule updatesymfony.php?q=node/1.symphony.php. (Explanation here.)core/includes/Drupal/Core/DrupalKernel.php. (Explanation here.)Etc.
Need concrete examples of how this will affect sites
I'm a software engineer, I know about patterns and architectures, yet I still have a hard time figuring out how this will affect Drupal and the sites we build with it. Don't get me wrong, I'm all for progress and improving our crufty API, I just think that concrete examples would help.
Not being a Drupal guy but an
Not being a Drupal guy but an interested party my understanding of how this will impact Drupal is:
1) much cheaper bootstrap (which is great on its own, but also makes point 2) all the more appealing)
2) ability to cleanly separate "blocks" (aka sub requests) so that they can be loaded asynchronously (ESI, javascript) transparently, which is one of the keys to enable caching as aggressively as the content permits (aka for example cleanly handling personalization)
3) untangling the core architecture (caching mentioned in 2) is just one of the positive side benefits of this)
4) less code to maintain and opportunities to make Drupal a citizen inside the PHP library ecosystem
Initial thoughts
While the anonymous function approach takes a while to get used to, I like Pimple.
And I like the fact that this is tackling the elephant in the room, and that is bootstrap.
There's a big question here, and that is: how do we actually arrive into a world where a full bootstrap doesn't mean loading all modules? Modules rely on other modules being present, changing that breaks everything. We can make it so that dependent module files get loaded, but that will still mean half of all modules will get loaded (thinking of a Commerce module, that then drags in Addressfield, Views, ctools, etc...). Better than nothing I guess.
Also a question of how this fits with Menu API. Will just extending DrupalControllerResolver::getController() be enough?
At first glance, the patch is grokable (though not through just reading the code, I needed to read this page in paralel.). But I'm afraid that it comes at the expense of a huge chunk of Symfony code that we'll ignore until it bites us (and when I say we, I mean myself). I like the idea of co-operation, though I'm not sure having the ability to embed other apps is actually that cool. Sure, it's running, but it's not using my db settings, doesn't access my users, I can't alter anything, it's basically a little black box.
I can also agree that we need info on how this will affect contrib modules, and how they could use it.
We have a sparse "How", but we also need a "Why" so that people are able to actually get excited.
To arrive into a world where
To arrive into a world where a full bootstrap doesn't mean loading all modules, I think the answer is using the Dispatcher as a way to implement the Observable pattern as a mean to replace our hooks (problematic because we need to load all code then we parse it all for each
module_invoke_all). And replacing procedural code in modules with OO code. As such, the controller doesn't have to care about which module is loaded, because if the module is a class, it will be auto-loaded/lazy-loaded at runtime. So the only code loaded would be the one we need for the request, nothing more. The exact implementation needs to be done though (this is my understanding and my idea, not a battle plan).Sounds good
I spent the last 6 hours reading this writeup, looking at the new code, stepping through it with Eclipse's debugger, trying to fully understand it and what its adoption would imply. I liked many things I saw.
Overall I think that if this is where WSCCI was heading from the beginning, adopting an already existing, tested and popular solution makes a lot of sense. My main concern is about the cultural shift this choice would imply: as @chx says, lots of parts of Drupal would simply not be Drupal anymore. It's not like replacing most of node.module with code from entity.module, it's accepting new coding patterns, best practices and even coding standards as part of Drupal. I'm not saying it's necessarily a bad thing, but surely it's a big one: many of us would need to learn lots of concepts from scratch, many more than what a new major release would imply.
To sum up I guess this choice would impact on the (developer) community at least as much as on code, hence it needs a huge number of persons agreeing on it to be carried on, much more than the relatively small number of developers that fully grasp the bootstrap and routing processes.
Another concern I have, but this has probably already been addressed while deciding to adopt parts of Symfony2 components, is that we would be somehow "losing control" over crucial parts of our codebase. Should we need to change parts of the HttpKernel component, for instance, we would be tied to the directives of another project for everything concerns timing, backward compatibility policies, release scheduling and so on. There could be situations in which forking would be the only viable option, altough being the worst one can think of. OTOH accepting such a decoupled system as the base of our architecture, would let us swap it with another implementation, should we ever have such an extreme need, hence I think we should be discussing the architecture more than the actual code.
That said, I have a couple of remarks about the new stuff: although the Symfony components used here seem to be all about allowing to implement a REST server, basically turning a request into a response, I think we should try to limit the presence of these concepts in our code. I think most of it can be rewritten (with very minor changes) without needing the concepts of request and response at all: context and result could be two viable and more neutral alternatives. IMHO this would make easier for developers to untie themselves from the standard page building concepts and think to their code as a standalone isolated entity, not necessarily tied to a particular workflow. In this sense I find the current proof of concept code a regression wrt the proposed Context API.
One last thing about context as a quasi-global: I think the approach #3 should be our ultimate goal, since switching/changing context without the stacking capability of the Context API might reveal problematic with a semi-global access to it. For instance, the multilingual initiative might rely on context much more than what @Crell hinted above and, without a reliable way to change the current context and restore the previous one afterwards, very bad things might happen.
Disagree
Plach, the web is built on request/response ... why would we want to abstract away from that?
Inputting a request and outputting a response is a beautiful pattern, and it has been demonstrated to work beautifully (see the django tutorial for an example).
As for your suggestion to have a global accessor for the context, why? Globals are bad ... if a method needs to know about the request context, it should declare a dependence on it (via the methods signature). If you want to pass a different context, it's significantly easier when that function declares that it expects a context:
<?phpfunction myaction($request) {
$new_request = clone $request;
$new_request->querystring = array('q=>node/1/edit'); // tricked you
someotheraction($new_request); // there's no magic here
}
?>
You can see how this is much nicer than this:
<?phpfunction myaction($request) {
$new_request = clone $request;
$new_request->querystring = array('q=>node/1/edit'); // tricked you
set_global_context($new_request); // hope nothing breaks
someotheraction();
set_global_context($request);
}
?>
Just a note that's not
Just a note that's not Plach's suggestion, that's the current proposal at http://drupal.org/node/1233232, except that rather than a single global context it keeps a stack so that context is only altered 'globally' within function scope. The idea being that this would bridge the system to the large amount of procedural code in core that doesn't take $context as an argument.
Misunderstanding
I'm not advocating moving a way from this pattern, just renaming the objects taking part in it: having request and response is no different from input and output or context and result, but it somehow implies that the output will be served. And since blocks might be nested, having a response object as result might be slightly misleading and encouraging possibily wrong assumptions.
Moreover any other context-dependent (reusable) piece of code might be so abstracted from the request-response workflow that having a request parameter might not make any sense, whereas a context parameter would be always fit.
Again, note that I'm talking only about the names of the objects not about their composition. Note also that I think that request and response are perfectly ok in the Symfony code, since it deals directly with the REST workflow.
There has been a misunderstanding about numbers, quoting @Crell:
I'm saying that passing the context around as a parameter should be our ultimate goal, although in the process we might want (as the current proposed Context API does) to adopt #2 to have an easier transition, as @catch notes above.
Actually thats a key point of
Actually thats a key point of the clean separation:
Every sub request can be turned into a "master" request if needed at the discretion of the developer without having to change a line of code that handles the sub request. This is one of the key things this proposal is achieving. This way developers can implement the entire logic without for example having to enable an ESI reverse proxy in local development. But if a reverse proxy is setup on the staging/production server and the relevant cache headers are set, then it can use an ESI reverse proxy. In the same way the developer can decide that the given sub request should be done by calling the controller via javascript asynchronously after page load etc.
So it makes absolute sense for sub requests to return a response.
response or context?
I had exactly the same though. While there could be a separate request to render a certain block (once we reach our goal), that's probably not the developer writing the block thinks about it. He thinks about a block displaying some information on a page, a block which depends on some context (like a node) to render.
Thus, +1 for keeping the context terminology. It can boil down to a request/respone in the beginning/end though.
Great!
From a quick read-through it looks like this would bolster Drupal's architecture, making it more REST-ful, and perhaps even more recognizable for those developers coming from MVC frameworks.
Drupal should evolve with the web.
And yes, rather than yet_another_api_call() in some_other_module it would be nice to have $context right in a request object in Drupal core. Good stuff! :)
sethviebrock.com
Reasons to get excited
Couldn't agree more, sviebrock. This is definitely what is happening in all web programming language AFAIK (other PHP frameworks, Ruby, Javascript - client side and server side). I really hope to see this happening in Drupal as well.
@bojanz, you asked for a reason to get excited. Here's one :) If we could achieve WSCCI by Drupal 8 and not Drupal 9 (aka sooner rather than later), it will make Drupal even more disruptive. For now Drupal is a CMF (Content Management Framework) we all use to create a custom CMS for our needs, that generates as many HTML pages as we need. Cool.
Imagine that WSCCI could allow us to build REST Servers just by clicking our way through the UI, and then serve HTML pages as well as XML/json for others websites, mobile apps, a front-end written in javascript (I'm talking about you Backbone.js), well anything. This would make Drupal the first REST Server Framework (RSF ?) that helps us create a REST Servers. The CMS we build on top of it would just make use of the REST server behind the scenes and transparently for the site builders/end users.
Put the other way around : Drupal 8 would allow any mobile app developer for instance to create a REST Server specific to its application without knowing NOTHING about PHP, and still be able to fully manages its data, configuration, and the entry and exit points of its web services.
Super cool !
Note that I'm pushing for Drupal 8, and not waiting for Drupal 9, because by Drupal 9 we would lose this innovation (the REST Server Framwork thing). Why ? Because the Symfony CMF is in the making. It's basically Drupal built on Symfony2 (including PHPCR and Doctrine). Please everyone, check their introduction slides, their 8 first slides explicitly mention Drupal, because they compare their CMF and ours, and want to make something better than us. Competition.
It should be noted that I am
It should be noted that I am sort of the "founder" of the Symfony CMF and original author of these slides and I am at the same time the Symfony "liaison" to Drupal. So one option is that I am trying to get you guys to fail, that I see Drupal's work as part of this initiative as complementary or that I am hoping that one day you guys will make Symfony CMF redundant :)
Dispatcher
thank you for this post. it makes me very happy indeed to see drupal moving the way of reusing code. i have seen too many good cms systems with the not-invented-here syndrome...
i just wanted to point out that if you keep the RouterInterface in drupal, it would be very easy to use the ChainRoutingBundle to tie all sorts of routing systems together and make integration into symfony a breeze. maybe take a look at the DoctrineRouter - it is the first attempt for the symfony cmf to read routes from the database and determine which controller does handle it.
Symfony Docs Update
Good feedback on the missing details about the attributes property in the Symfony docs Crell. We will ultimately have standalone documentation for each of the components (whereas the current docs are mostly aimed at the full-stack framework), but I've added several additional details about the Request object
https://github.com/symfony/symfony-docs/commit/4b734749b09b35f5911eaef97...
Really fantastic, detailed post - these problems are shared amongst everyone - from flat PHP developers, to Drupal devs and custom full-stack framework devs. Really happy to work with the Drupal community, in whatever capacity is decided.
Thanks!
Great
Crell, great write up and it sounds like we're heading in a great direction.
I think option #2, passing the request around, is definitely the best direction. Globals are bad.
Symfony is heavily based on concepts from Django (a popular python framework), so anyone interested in playing with a beautiful and mature full stack request/response implementation would enjoy from going through the django tutorial.
Indeed I was discussing #2
Indeed I was discussing #2 with Crell. This is the best solution to really ensure that sub requests do not rely on data in the parent request that isn't explicitly passed to it from the request in a serializable way (to enable ESI/async js in a transparent manner).
The approach Symfony2 took with placing the request instance into the service container but using service "scope" scopes for "hiding" previously created instances of services that rely on the request inside sub requests is likely going to be too confusing to beginner Drupal developers.
Responses
Wow, no ninjas? I'm relieved. :-)
In any case, I'm going to respond to a number of comments at once here, since there's lots of overlap.
1) Yes, this is a big change. The closest thing I can recall in Drupal history would be the Drupal 4.7 FAPI patch, which necessitated rewriting, um, most of Drupal, in one massive patch and then spending the next 8 months cleaning up everything that broke. My hope is that with Git, testing, and proper temporary BC shivs we can pull it off, but I fully acknowledge that this is not a one or two week task. That's why, if we're going to do this (and I believe we should), we need to collectively commit to it.
2) I considered inlining some code samples from the branch into this writeup to help the descriptions. I decided against it because one, this post was getting obscenely long as is and two, I wanted to get it posted before midnight and I was tired. If there's any particular parts that still don't make sense after getting the code and reading through the walk-through above, let me know and I'll try to clarify.
3) I didn't really deal with hooks in this post, because I don't quite know what to do with them yet. As chx and others noted above, hooks are a form of Aspect Oriented Programming, which is a weird offshoot of event-driven programming. Such a model requires an index of every event/aspect/routine to call at a given point. That's a hard-requirement for any such system, and building and maintaining that index is potentially expensive. With hooks today, we take advantage of a clever hack of the PHP runtime and use function_exists() as our index. As of Drupal 7 there's some caching and lazy-magic-naming happening on top of that, but it still comes down to that. We're at a point, though, where I don't believe that scales.
However, replacing it with something else requires figuring out the indexing problem; we need a way to detect what responders are available to a given event/hook, do it fast, have it fail gracefully (unlike the class registry), and it cannot rely on loading code into memory for all indexing because then a rebuild requires a massive memory ballooning. I don't yet have a solution to that. It is, however, orthogonal to the page/response building flow that we are discussing here. (Or, it should be and if it's not, there's a problem.)
4) Speaking of hooks and other existing code, I agree that "everything takes a request object" is a good end-game. However, we weren't pushing for that in Context API yet because, frankly, we didn't want to be quite that ambitious, especially in a single release. I still don't think we can bank on being able to do that in all one fell swoop. Even if we do, we'll need a BC shiv during development because we cannot leave core in a broken state for 9 months straight.
5) I don't really see a value in "hiding" the request/response nature of the web or a block. As in the code here, in the typical case we can still have the "do something" method of a block/controller return a string or render array, and let the parent class turn that into an actual response object. (Inheritance, for all its architectural issues, is one of the very few performance freebies in PHP.) As Lukas notes, though, having each controller able to return a response object itself is necessary if we want to be able to render blocks asynchronously, in separate requests. Which we do.
6) As far as site building goes, I foresee it being closer to a Page Manager / Panels / Context Admin style approach. That's an approach many are already using, as it is very powerful, but the UI is currently kinda crappy. We would need to have solid UX work done to be able to leverage it properly (whereas something like Symfony relies on just hard-coding a one-off instance of a "page", because it's strictly a framework, not a site-builder-facing CMS). I'm still trying to assemble a dedicated team for that. Core would then ship with a default set of pages that essentially mimic what we have now. However, something like OpenAtrium should be able to easily rip that out if it wants to. Basically what core ships with, then, is Snowman.
7) I don't know all of the impact on module devs yet. I haven't thought through all details, just enough details to be able to present it for discussion. I don't want to, nor could I, think through every detail myself. In general, I would expect most modules to be a combination of:
Keep up the discussion, folks!
Thank you
This helps clarify things. I realize asking for code examples is a bit difficult based on, as you say, a weekend's work.
Regarding this:
And this:
How much of a BC shiv are we talking about, here? How feature-complete and how robust? Is it only "temporary" or should we aim to provide procedural wrapping so that the codebase remains recognizable to your average Drupal developer? (I realize these are very vague and open-ended questions, but I don't recognize the module Crell describes here. I feel I'd have to rewind back to 2006 and start learning Drupal all over again.)
I see a lot of rhetoric in some comments above. I'd like to set that aside for a moment, and simply postulate that the community decides this is the best direction for Drupal architecturally and as a project, and think about what happens next.
I don't mean to spread FUD, nor claim that it's not feasible, nor reject the architectural change because it's not "Drupally" enough. However, I do want to make sure we keep the impact on developers who aren't participating in this conversation in the foreground, and consider what we can do to make sure the codebase is recognizable and comprehensible to the people who currently rely on it.
Good questions
One of the reasons for this post is that I don't know the answer to many of them, yet. Also, given how big and varied Drupal is once you mix in contrib, the module I "describe" above would likely be very recognizable to some people, moreso than core modules are. In a sense, if you're used to writing new Views plugins or using OO-based ctools plugins, you're probably closer to this model than you are to current Drupal core. That is, of course, not everyone.
To an extent, one reason that I'm currently in favor of drupal_get_context() (or whatever its equivalent would be in this model) is exactly that: It means we can leave stones unturned and the system still runs. Similarly, unless there's a very convincing performance argument for it I'm sorely tempted to leave hooks as-is, for the most part, for Drupal 8. There may be a convincing performance argument for it, I grant, but there's also serious challenges to gutting that system, too, as I note above. (Mostly relating to an indexing mechanism that doesn't suck. As of yet I don't know of one.) But that at least is something that can be comfortable for current developers.
Also, my intent with the plugin push (shot 2 of WSCCI) is to unify a lot more parts of Drupal. So while there may be a lot to learn, it's at least the same thing to learn each time. Once you understand how one piece works you understand how nearly all pieces work, just with a different PHP interface on the object.
I have also been pondering if we want to keep around page callbacks, as a sort of "degenerate case", at least for one version. That is, if you have a page callback defined (albeit perhaps in a different way than we do now with hook_menu) then there's a default layout configuration that core supports that has a "whatever the page callback says" block in it. That lets you mimic the "I have a form, put it at this path, I don't care about the rest of the page" workflow that is common now. (Whether we do that with page callbacks as now or something else, that's a common enough workflow that we need to make it simple one way or another.)
Similarly, I expect that getting a DB connection via the services object would become the "preferred" approach, but we could keep db_query() and friends around for a version, just mark them as deprecated. You couldn't do really solid unit testing with them, but contrib maintainers would have a version to get used to it. (And coder_update would have an easier time of it, too.)
Then we can eliminate many of those deprecated wrappers in Drupal 9, while doing other cleanup. (Since I'm 100% certain that if we go this route, there will still be stuff we need to fix in Drupal 9, because there always is.)
Anyway, those are my current thoughts on the subject.
Performance/dx/transition trade-offs
I think plugins seem complicated when you first look at them but actually we've been using them all the time. We just call them hooks and don't provide any metadata around them. Once you learn them it's fine, but it's easy to forget that AOP is pretty opaque on first view. Less is not more, in this case.
So, it's slightly off topic, but could the plugin model also be applied to hooks. In other words, you specify all your "hook" callbacks in a single hook_aspect_info() function that builds a hook registry that remains cached. The following example contains an aspect for hook_menu...
<?phpfunction mymodule_aspect_info() {
'menu' => array(
'file' => 'includes/my_menu.inc',
'function' => 'mymodule_declare_my_menus',
'weight' => DRUPAL_ASPECT_DEFAULT_WEIGHT
);
}
?>
This might give us some other benefits as well, such as not having to change module weight just to fire hooks in the right order. Could plugin definitions also come from the same function?
Edit: This could also be done using a class, as per crell's examples...
<?php$aspects = new stdClass();
foreach (module_list() as $module) {
$class = "Drupal\Module\{$module}\AspectInfo";
if (class_exists($class)) {
$aspects->addAspect(new $class());
}
}
?>
This is not an "indexing mechanism that doesn't suck" from a performance perspective, but it has some advantages over hooks as they stand, including the potential to morph them into plugins, and provide a bridge to 9. Maybe this will spark some ideas from someone else too...
Pages could well be provided as a plugin type as well, which would allow the menu system to simply reference a plugin rather than a function callback.
Edit: I'm happy to prototype this after new years if there is any interest...
Hooks are not plugins
As I noted above, I kinda want to avoid dealing with hooks right now. They're actually separate from the flow we're dealing with here. What we'd be replacing is hook_menu specifically, and hook_page_alter. twitch
Also, plugins are not a replacement for hooks, nor were they ever intended as such. Plugins are a replacement for the dozen different quasi-plugin-like-things we do in core and contrib now, in a dozen different ways, only one of which is "things that look like hooks and are documented as hooks even though they're not hooks and we're just lying to people to make their lives harder" (eg, Field API "hooks"). Those would get replaced with plugins. hook_boot, hook_node_insert, and so on would be unaffected either way.
Plugin prototype code already exists in the wscci sandbox, including a preliminary implementation of Field API using plugins courtesy of yched. (Sweet!) If you're interested in helping out there, contact EclipseGc or neclimdul as they're really the point people on that at the moment.
We already have a hook to define metadata on hooks: hook_hook_info(). It went in very late, though, so it's not widely used and catch has been reticent to commit patches that make more use of it.
The second example above is essentially what the Symfony EventDispatcher component does. In fact I'm already doing exactly that in reduced form for the service registration in the wscci-symfony-dic branch (discussed in this thread). The main difficulty is that class_exists() triggers autoloading, so you have a lot of cache misses, plus you have to instantiate the code in order to register anything. That's why it fails the "doesn't suck" test.
Plugins and hooks are not really germane to the router discussion, though (aside from controllers probably being plugins), so let's leave those aside for the time being.
Your code suggests
Your code suggests registering hook invocations, and both chx and xjm have raised it, so I was postulating a method for making Drupal 8 work like Drupal 9, but still be groked by people familiar with Drupal 6 (because there are still lots of those people around). Fair points though, I agree its off topic.
Don't worry Larry, I'm taking
Don't worry Larry, I'm taking up a collection for hiring the ninjas so this won't go on too long!
Can I start by saying that the DX for Drupal 7 is many times worse than for D6 due to the added layers of indirection (e.g. entity controllers) that make it hard to understand WTF a seemingly basic API function is doing. What I'm reading here make me concerned that we might as well be re-implementing Drupal in java and utilize the servlet API since it will be that inaccessible to the current Drupal community.
I also quite disagree the comment above that we are always working with a request and response. In many ways that's the failing we are trying to get past. If I run a drush command is there a request/response? If I'm unit testing a subsystem? If I'm getting the content of a single block? For that reason I do think something like context/response is more appropriate as suggested.
Perhaps we should indeed be writing code that looks more like JavaScript or python and pass the context into every API function as the first argument? That might have some interesting implications - for example, we might (deliberately) reduce the ability to hack things at the theme layer by not giving it the context and hence making it hew to just a templating/output responsibility. This would also mean we don't need to have any globally accessible context nor any stack per-se.
In any case - let's take a step back here. If we are doing this right, we want to try to make the low-level underpinning both robust and performant, and also make it so they are not in the face of the typical module developer NOR visible at all to the site-builder, but only visible for those who dig deeper. Is symphony2 really the right choice for a rewrite? The discussion above makes it sound too flexible and with too many existing levels of indirection due to it being a generic framework.
If you are working with
If you are working with Drush, you have a Request/Response: Request is what the user typed, with all arguments and stuff; Response is what you send to the console. It's not different at all from the HTTP Request/Response, only the protocol and transport layer changes.
Pierre.
Yay, ninjas!
Reversing the context/request does have an interesting implication for Drush, I agree. As I have it implemented right now, the context property does have a reference back to its request, so we could pass the context around... However, we still need the request any time we're passing to a new Kernel, or anything else that is expecting a request, not a Pimple object.
I think I agree about "we want to try to make the low-level underpinning both robust and performant, and also make it so they are not in the face of the typical module developer NOR visible at all to the site-builder, but only visible for those who dig deeper." However, that statement could take many many many forms. I don't think we can make the changes we need without a non-trivial impact on module developers. Even just turning page callbacks into classes and doing nothing else would be a not-small change, but a necessary one. Similarly, you cannot decide how to construct a page design unless you have some clue about how Drupal builds a page. (I've seen pages built by people who didn't. It's not a pretty sight, nor a pretty site.)
Certainly I wouldn't expect the majority of module developers to be digging inside the kernel, just as we don't expect most developers to be digging deep into the database condition compiling algorithm. However, understanding at least to some extent the concepts that are happening under the hood is important to being an effective module developer; that's true now, and it's true no matter what we do in Drupal 8. Those concepts should be as simple to understand as possible.
Even now, I wager most core devs don't have a clue how the D7 page rendering process works. I don't, and I've tried to step through it. This model would have far fewer exceptions, edge cases, and "things happening sideways". That makes it easier to grok, I'd argue, as well as more flexible.
Resistance
Since I didn't start using Drupal until 6, I'm curious of the level of resistance there was when the form API was introduced. If it truly was an 80% rewrite in one patch then I would have expected the same level of resistance. Or has our community become so fragmented that we can't agree on anything anymore.
I haven't formed an entire opinion yet. Reading this thread (along with the several others), the "supporters" seem to be using more evidence based reasoning with code examples and patches. The oppressors of change seem to be just speaking in more abstract concept using terms like "DX" without any further explication or alternative suggestions.
Neil Hastings
http://twitter.com/indytechcook
There was a reason: security
The biggest reason for having Form API is security. CSRF, for one, option checker for two. http://drupal.org/drupal-4.6.2 this scared the bejeebers out of everyone involved. Also, you couldn't find any dissent (aside from it stretching the release cycle so much). The community was much smaller then and during the form API conversion a significant portion of contrib got converted as well.
Thanks
Thanks chx. I was just curious.
Neil Hastings
http://twitter.com/indytechcook
DX
The reason why developers are concerned with the topic of Developer Experience (DX) is that remaking Drupal to use a different manner of implementing hook_menu brings about a key and fundamental change to how developers currently author contributed modules / custom functionality / get paid by our clients. As such the prospect of throwing away all the knowledge we've acquired in order to relearn those how to solve those problems a different way brings with a hefty amount of anxiety, fear, uncertainty, and doubt.
How much of Drupal will change as a result? We don't really know.
How much will what I know about making Drupal sites be relevant in the future? We don't really know.
How valuable will it be to stick with developing with Drupal in the long term, or should I devote my career and my time to some other framework? We hope you continue with Drupal but can't really provide any assurance that you are indeed investing your time wisely as we have a history of breaking things from one major revision to the next.
These are the questions that any multi-discipline Drupal shop have to answer when resource planning for the future. And with it comes a few other hard questions.
Today Drupal is awesomely popular because its provides solutions for a unimaginably large set of problems when building websites. In my view, Drupal has become the general problem solver for nearly any kind of website because of the hundreds to thousands of developers how have started with zero knowledge and either by their own or through the help of the community have added functionality that was previously missing. The questions:
To answer these questions we need to either know the future or have such a deep understanding of how the developer experience will be as a result of all of these changes that one can clearly state that "It's all going to be O.K."
I'm not sure if anyone here knows enough about the changes stated here, and their natural consequences, to state that. And until we are at that level, we're not going to have consensus on this set of changes.
Here are few next steps that I think would be helpful for getting us there:
1. Make a few simple modules that use the new Symphony based code
2. Port a few existing modules to use the new Symphony based code
3. Show the before and after for those modules and discuss
Software Engineer @ The Nerdery
Good questions
When mixing business previsions and Drupal evolution, I would ask another question :
If we don't implement at all the WSCCI initiative (whether with Symfony or not, it doesn't matter), will Drupal lose the advantage in the CMS space because Drupal will, by the time Drupal 8 is out, ship, at best, with a deprecated functionalities, and at worst, a big code debt if the clean-up is not properly done ?
My answer : it will.
So, your questions about DX are completely legitimate, because they concern a big part of our community : a lot are Drupal site-builders-who-write-code/contrib-developers/custom-developers who interact with Drupal the software through its API, so a major API change definitely has major impact on the community. However, if it's just a matter of API, don't worry : this big shift will require big documentation, that people will be VERY happy to give. Furthermore, they WILL provide it anyway, because core devs are VERY conscious that contrib rely on their work.
Answering your questions :
1) I already answered it. API change is not the big threat to Drupal, but not evolving with the web IS.
2) and 3). Hopefully not. Contrib modules maintainers are knowledgable people, who will make up with the new API and the new "Drupal way". And this new "Drupal way" will be, I'm sure, appealing enough to see pull new developers in, who will help surely create new modules, and even help maintain previous work (aka Drupal 6/7 work).
As for creating a module to show how to develop with the proposed new "Drupal way", nothing has been committed to core, even agreed by the community, so it's way to early because there's nothing to show yet. But this new "Drupal way" will surely see the conversion of core modules, which will be very good examples for contrib people. And all Drupal Cons/Camps happening after it's committed will be plenty of occasions to explain how to develop against the new API.
If the cleanup is not done?
What I am stressing is this: Drupal 7 is awesome. It needs cleanup. It doesn't need a rewrite. Not evolving with the web? We are doing a mobile and a HTML5 initative.Edit: I wish I could delete all my comments but that I won't do. However, consider them deleted. Do not quote them or credit me with these opinions. They have been obsoleted.
Problem comes with scaling.
Problem comes with scaling. Full drupal bootstrap is expensive and both initiatives benefit from this one.
You can't anyways
You can put lipstick on a pig (make Drupal a little faster) but PHP is dog slow no matter what. Everyone is scaling by not running PHP. Yes, at a time we were excited about how PHP is a shared nothing architecture and scales wonderfully by adding more machines... but that excitement went away when evented languages and webservers served magnitudes more requests from the same iron.
I am not saying we shouldn't make Drupal faster but just making Drupal boot faster will achieve ... nothing (rounded to the nearest nothing).
Edit: what I am trying to say... and I said so already: Drupal will never be a first class REST server. You would truly need to rewrite it from ground up to be even a first class PHP REST server.
I'm tired of seeing this
I'm tired of seeing this argument everywhere, PHP may be slower than other programming languages, depending on how you benchmark it: that doesn't get rid of the fact that I currently have a D7 with only core on my box, right now, displaying 10 nodes on the front page with 10 fields each, generated with devel_generate, and I end up with more than 200 (sometimes 250 seems to vary) cache queries along with something like 30 SQL queries when I hit this page. I'm quite sure those 250 queries run 10 times slower than the full PHP code run. The worst bottleneck ever is not the language itself, but pure software design.
EDIT: That said, I leave two you alone to this wonderful debate about scaling.
Pierre.
Couldn't say better.
I'll just add that people ARE writing "request-response" based applications, most often following an MVC pattern, with other PHP frameworks, with Python, with Ruby, with Java. They didn't wait node.js and the like, because this is how the web should be working. They just understood it before us.
Well one of the key thing
Well one of the key thing Crell's work enables is to more easily generate blocks separately for caching and/or asynchronous loading. Of course separating blocks like this only makes sense if bootstrap overhead is reduced, as when doing such separation you effectively end up having more individual requests each doing less. But this kind of caching enables exactly what you are talking about: having a dynamic php website, where for most requests you don't need to call any php, as the reverse proxy can do all the work (well and for those people who cannot install varnish etc, we can at least offer them a reverse proxy written in php, which again would side step the entire bootstrap).
This has been stated many
This has been stated many times by both Crell and others, but I think people are overestimating the effect of this approach on performance, and underestimating what's required to be able to have a 'light bootstrap' to the point where you can serve a small chunk of HTML very cheaply from a dedicated request compared to just doing it inline as part of the page (in terms of overall time vs. resource usage).
Leaving hooks out of the equation for the moment (as has been requested above), if we want to make the code weight of a full bootstrap lighter, then we'll need to move a lot of code that's currently procedural and scattered around /includes and system.module into PSR-0 compatible classes and lazy load them. This is already happening (albeit fairly slowly) in issues such as http://drupal.org/node/1225404 and http://drupal.org/node/81461 are part of that process as well as several other issues linked from http://drupal.org/node/1224666. That refactoring can happen orthogonal to whether there's a complete rewrite of bootstrap - we'd just be gradually slicing bits of it out until there was very little left. I don't think it has any relationship to the proposal here, except that a conversion to Symfony2's HttpKernel (or lithium or some other comparable system) would make it a necessity. In other words, it's not an argument for a rewrite like this, it's something that can be done anyway and that's already in progress. The reason to convert those systems is to disentangle them, for unit testing etc, not really for performance (if we can save 3ms or so loading code that's a nice side-effect).
The main change to this in terms of wscci is removing front loaded globals such as $language and replacing them with a single request object that lazy loads data, but that's already handled in the existing wscci patch as well.
However, the code that is loaded during full bootstrap, even though there's a lot of it, is not really the problem in terms of performance. If you use an opcode cache, then require_once of a dozen or so files is pretty cheap. In fact it may be a wash, or slightly faster compared to loading less code, but doing so via a PSR-0 autoloader (I haven't benchmarked it yet, but there's an open issue in http://drupal.org/node/1241190 where those benchmarks or some profiling would be welcome).
I just took a standard install of Drupal 8, created one node, then profiled the front page. The full uncached page render to 60ms.
Of this, 20ms was spent in drupal_bootstrap() inclusive. Over a third of of that was spent initializing the variables system, which has nothing to do with loading code at all. Being able to lazy load things like the $language global will definitely help to reduce some of that further, but it's generally pretty small pickings there.
So if we want to make Crell's example of loading a node and rendering it to JSON, where's the expense in that? It's not in loading form.inc, graph.inc or theme.inc. It's in loading entity info metadata, module metadata if there are a lot of modules installed on a site, field info (see http://drupal.org/node/729186 and http://drupal.org/node/1040790 for background). The expense is not so much about bootstrapping Drupal in terms of /includes, it's about bootstrapping subsystems such as entities, fields and the module system, which you will now be doing for more or less every ESI callback instead of once for the request. And this expense is in loading data/configuration, not code.
This isn't really a comment on the proposal itself - there are advantages and disadvantages to it other than these points, but if the main priority is to reduce code weight for 'performance of ESI callbacks' then I don't feel people are properly examining what the actual performance issues preventing fast ESI callbacks are at the moment. Even with lots and lots of modules enabled, it is often still not code weight.
In terms of evaluating how much impact this could have on the 12ms or so that's spent bootstrapping core, is there a decent Hello World example from HttpKernel and Lithium we could compare to say doing print 'Hello World'; drupal_exit() in a page callback (or extra points for using a delivery callback to avoid the ghetto print statement). Profiling between the three of these and having the code available to compare and contrast should also provide useful data to people on how different they are.
I certainly agree with you
I certainly agree with you about cleanup! This kind of major change also come potentially with a lot of old code removal, which seems to do some of the cleanup for free in the end. I don't agree that D7 is awesome thought, but that's a personal opinion I guess.
I am more than eager to see menu handling becoming a router in a request/response driven software, that means we could isolate some of the Menu API madness more easily and make it evolve slowly. It also mean that page rendering functions would get out of this infamous 7000+ lines common.inc, and it would surely remove all block handling code from core's includes and move it into a nice encapsulated place. Why do the block and region handling are in a common place anyway? It's not common, it a specific feature!
Pierre.
As for creating a module to
I find the idea that one would get an API committed and only then write the implementations against that API to be quite troublesome. Not only does it make it much harder for others to get involved and evaluate the ideas, it also flies in the face of all current API design best practice (for example, Joshua Bloch's slides on How to Design a Good API and Why it Matters).
The only way to verify assumptions about the API is to write against the API as you develop it. I agree with xjm, sun, bojanz and cosmicdreams, we need to see examples, both for community engagement and for better API design.
This is also a troublesome idea. We can't just say "we'll get to that later". That's why so much of the Drupal Handbook is still Drupal 6. Even our most important handbook, the security handbook, still needs to be updated and doesn't have key changes from D7.
We need to document how to do things the new way as we go along.
We need to document how to do
Yes!
Donna Benjamin
Former Board Member Drupal Association (2012-2018)
@kattekrab
While I agree that examples
While I agree that examples of code are good for people to see what's really different for them for an "API user" point of view: this is kind of hard to provide at the state the code is right now, for multiple reasons:
What's missing is maybe some explaination of what it will allow us to do (and it allows to unlock a lot of existing core problems), if we go down this road. I think this has already been answered several times in the various WSCCI group's posts, but they didn't get much of people's attention except for those working on it since the beginning (but they are public still).
The goal of this Request/Response bit of API replacing some bits (not the full) bootstrap is to allow us to extract pretty much every bit of the page as controllers, allowing us to render pretty much anything as "standalone" components in their own HTTP thread which allows to take great advantages of ESI gates and such. The use of the Synfony 2's HttpKernel also open the door of using their ESI component and others they already have developed we may need.
When catch says that today, bootstrap can be quicker just by enabling lazzy loading of components (he took as example the caching and locking layers) and that we don't necessarely need this PoC to make it work, he's right! Nevertheless using a Request and a nice DIC also help us to centralise those components lazy loading and put them all in one single place where we can access them. But right now, it's not planned to replace the existing procedural API to access them: even if the right way to use them may become getting them from a storage container attached to the request (I can understand this frighten some people, said like that) the "Drupal Way" procedural functions can be kept as proxies to this storage container: the module developers won't see a difference in the end.
Pierre.
Just a word for the Drupal
Just a word for the Drupal community : we're very lucky to have Symfony2 core dev and Lithium lead dev participating this very discussion. Let's get the most out of them to make Drupal 8 rock :)
Making Core Maintainable
This proposal is about making core maintainable as much as it is about web services. So we need to evaluate it in terms solving issues listed here. It will have other major side effects in module development, but the hope is once we fix core, these other problems can be solved without trouble. People understandably are skeptical about the last statement. May be the proposal can give some more clarity in to this through more discussions. We are yet to see an alternative vision for core which does it with lesser effort. This proposal also has the advantage here of using existing, well tested, documented code.
Dev Community Growth or not
@cosmicdreams
I am glad you bring that up.
A gripe many developers have with drupal is it is using such a different concept than other software.
People simply dismiss it because it is not OOP.
Concerning that: even if an AOP and an OOP implementation would be technically equally efficient for our tasks, it would be an advantage to not alienate new developers by having to learn a whole new concept.
So the point of: will it drive more Drupal Devs away because things change heavily or will it make easier to pull new ones in because they find familiar concepts in our software is not completely a technical one but a social one and very important because of that.
If Devs have spent a lot of time learning concepts, they want to be able to re-use it in whatever system they are working in.
The same goes for using Symphony2: beyond technically looking at it if it is the best-of-breed: are there people familiar with Symphony2 and concepts used in it that will be more willing to work with Drupal if we re-use their Kernel?
What became evident with Drupal 7 especially is that core needs an army of maintainers and we need to grow their numbers. In the medium-to-long-run it will become vital to meet people half way by using concepts and elements they already know. So they can work on drupal but not having to re-learn everything. Being more of a web developer and less of a Drupal developer.
Does core get more understandable through the proposed changes or even more tangled?
Life is a journey, not a destination
Thanks, missing my other post
I wrote another giant post highlighting the risks and rewards of what is being suggested here that I don't see on the thread yet. Maybe I clicked the preview button instead of save, ugh. This post will be an abstract of that.
Essentially my point was that, I personally love the idea of making Drupal more MVC-ish + having DI because I come from a MVC background when developing non-PHP projects. But changing Drupal to have that capability sounds like we're going to have to build Drupal from scratch.
Is that what we're talking about here, that we'll have to rewrite Drupal from scratch based on these principles. If we aren't, what's not going to change?
Then I listed a big list of potential risks and rewards for the rewrite.
Software Engineer @ The Nerdery
Drupal is nearer to HMVC. If
Drupal is nearer to HMVC. If you're not a purist, you can already implement the typical web-MVC pattern to a large extent in Drupal within a custom module. (e.g. use an inc file for the model, treat the module file as your controller, and template for the view.)
Me too
I've been observing that Drupal's "unique" architecture is a liability as much as an advantage for some time. It makes it very difficult to bring in new blood that isn't a "weekend self-taught warrior" who doesn't know from anything else, and people who learn Drupal code then have a very hard time adapting to anything else. That's not a good form of job security. ;-)
If I am honest with myself, if I were just getting into web dev now I would not go with Drupal. And that is something that scares me.
Just to weigh in my two
Just to weigh in my two cents; I'm not a module contributor or code contributor in any sense, and Drupal has been the most amazing, awesome tool I have ever used. In fact, its so awesome, that without writing a single line of code, sans CSS, I have made many awesome websites. I would highly suggest people get into Drupal - I've run and continue to run user groups and spread the good word, so to speak. What we need to focus on, as individuals, is to re-learn how to learn, so we can learn new tools with ease -without guilt/fear/etc. Drupal is fine. A change is fine too!
What we need to focus on, as
Precisely! Indeed. Exactamondo.
Change is inevitable. Embracing change means learning how to learn and adapt faster. Drupal is hard to learn. Drupal has been hard to learn for a long while now. Making Drupal easier to understand, working on pathways, reference implementations, documentation and learning materials is important - but that also takes time and energy.
We all need to have a better idea of where we are going, then it's easier to embrace change, and dive into learning something new.
Donna Benjamin
Former Board Member Drupal Association (2012-2018)
@kattekrab
Ch-Ch-Changes
I'm with Drupal really long, but until this moment I haven't had time to dig deeper and become Drupal (contrib) developer. Hell, I don't even know PHP that good.
As I lost my job recently I have that freedom (luckily) to invest not only weekends, but all my time to something that I will learn and hopefully make some money from it - I decided to dig deeper into Drupal (that's why I'm following this discussion, after all).
Point is, yes, I really ask myself that same question, am I insane to learn all that crazy stuff about D7, just to throw all knowledge away for couple years, when D8 knock at the door?
Shitty situation, indeed, but, as I'm in love with Drupal and his (mine!) community, I will probably just stick around OR give up all this techno shit and run to assembling not web sites, but permaculture gardens. ;)
BTW, I enjoy reading all posts and the fact that people from other PHP frameworks anticipate - I want to believe that better world (Drupal?) is possible, and all this will prove it hope so. :)
P.S.
Maybe all this ranting about code / API changes and what it bring is useless.
In life, things change constantly, which is also evidently in web business. Ignoring necessity of changes and not working on ability to step out from your comfort zone IMHO is not good in the long run.
IMHO, concentrating to make change much less painful and much more understandable is good direction.
I started developing with
I started developing with Symfony since beta versions until now that it is version 2.0. The change has been great.
I started serious with Drupal with version 5.0 and seen it grow since there. And I must say that every day I build new sites more with Drupal than with symfony, even web applications. Because the quantity of modules, the easy of building prototypes quickly and the marvelous of distributions. Definitely, code wise, symfony is more beautiful and well organized than Drupal. But Drupal strength is its community and ecosystem around it (contributors and contributed features, distributions). Drupal is the linux of the CMS. And I would recommend any new web dev to take a look at it.
Uniqueness isn't such a constraint
I think a couple of issues that cause developer resistance are a) there is an idea that procedural code is somehow 'a bad thing' (though nobody criticises this of e.g. apache) and b) the other frameworks have all essentially grown out of the Rails approach, so 'web-MVC' (+ activerecord) has become normalised as an architecture. However, it's worth considering that despite being seen in many frameworks and languages, that architecture still hasn't delivered a generic popular CMS/CMF or comparable base of contributed modules.
Also, I'd consider that all existing architectures are starting to look dated in the face of developments like node, backbone, NoSQL, apps etc. While Drupal is already a great choice for a new developer in terms of getting hired, the D8 initiatives look good to me in terms of addressing technical gaps and make Drupal less 'particular'. But uniqueness isn't such a constraint in any case. Apart from Drupal itself, node.js and Salesforce provide very different examples of how a different architecture or domain-specific approach don't hinder adoption.
Not Web MVC
As for "web MVC", I have long been on the record as opposing it as fundamentally dumb. :-)
If anything, this is moving even further toward a clean PAC model; each block becomes its own controller (even has the name) which hits the datastore, renders stuff, and passes it back up.
I don't see what node.js has to do with it, though. It's a fundamentally different model of writing a server.
Cross-posting
Yep, I must agree, the PAC model (hooks are the glorious representants of this model) will get cleaner. It's just that PAC looks like MVC when looked from afar. Sorry if I brought some confusion with my comment below. (Crell, don't send ninja plz !)
Agreed
Indeed, but isn't that a major part of Drupal's architectural difference?
I was invoking node and Salesforce as counterpoints to any possible assumption that standardisation in itself causes or is necessary for producing a successful and thriving platform. (Not that I'm against standardisation.)
I don't think this is this
I don't think this is this uniqueness (not being web-MVC) that made the CMS popular and the contrib so numerous, but more the power given to the site-builders, and the ease of using the code (hooks are easy, really). WSCCI won't remove this uniqueness, Drupal won't lose its nodes, content-types, views, and WSCCI won't fundamentally break the site-builder experience, not even the contrib developer experience. A new API may replace another, but whatever the API it will still be based on the approach of hooks : "let me do my stuff when X happens" (X ranging "form being displayed", "node being created", etc.).
And wrt node and backbone : you're wrong. They don't change the web-MVC pattern. Node is just an environment, but the most popular web-development framework for node, Express (Express is the 4th most depended on node module of all kind, and the 1st webdev framework in the list), is clearly web-MVC oriented. You assign response to request based on their url. And backbone.js is a wonderful javascript framework I played with a lot, and it's definitely MVC-oriented (well, it's more MVVM, where the View and the Controller are the same, with bindings between the templates files and the view... but still MVC at its heart).
The new technologies doesn't change anything int the architecture design of web frameworks and applications. They bring other qualities, but web applications are mostly implemented using a MVC-like design, whatever the technology.
So, bringing MVC sauce will definitely help bring more people from the outside to the Drupal game, and help grow our own people to the most used design in the web (and make them better web dev). We should not fear this change, because it will have many benefits for the software and the community, at the cost of requiring more work, evangelization and documentation when migrating sites and people to Drupal 8.
I think of Drupal as already
I think of Drupal as already being web-MVC+ in that it is already PAC (so we don't need to adopt more of a web-MVC approach as we have been some way ahead of it for a long time).
wrt backbone and friends, it was a more general observation on how web architecture has been changing recently. The point wasn't that they aren't web-MVC (backbone itself explicitly is), or whether you define node as a software platform or framework, but that they can bring new integration patterns and requirements (e.g. consider how Solr drove Views to be able to connect to external data sources)
Inspiration from others
I'd love to hear your thoughts about the new patterns that we may bring in Drupal from node and Backbone, I never thought this way myself.
At a more technical level, I can say that, having played with Backbone, the pull-based and superblock approach proposed by WSCCI will allow to build a front-end to Drupal (I don't think the admin as we use it) using Backbone, and this is just a wonderful idea. There also other pieces of javascript software for the front end that were mentioned during the various discussions in gdo or the WSCCI sandbox (can't remember names now, but they were cool, really). All these new technologies were considered during the work of the WSCCI team.
What I mean is that WSCCI is trying to build Drupal in a way that either takes into account some new patters out there (Dependency Injection Container for instance), or takes note of existence of other nice tools so that they can work with Drupal (Backbone and all the collection of Front End MVC-like javascript frameworks : Sammy.js, Batman.js, Angular.js, Ender.js, Spine.js, Knockout.js, ... I made a presentation in a local Drupal Meetup about the new wave of javascript tools). But even without talking about javascript, it means you could build a dynamic website with Ruby or Python or other PHP frameworks, or GWT (sic!), or even node.js, using Drupal as a back-end to manage users and raw data (like a true CMS) !
THIS SIMPLY ROCKS. That's why I'm not reluctant to the change WSCCI will induce.
I wouldn't say that Drupal
I wouldn't say that Drupal suffers from not being an old-fashioned style CMS with separate front and back end. I've done that with Zend and D6 but that style is very difficult as you can end up trying to sync two software projects with a data source in the middle - something like trying to drive two cars with a hand on each wheel. But we can certainly benefit from Drupal CMS being built on an API that doesn't require posting forms to add a node :) With the web services module, you can already add extra clients for specific tasks (looking at doing exactly this with backbone). And before web services you could already do things like use Flickr as a Drupal client with XML-RPC.
The really interesting new bit will be using Drupal as both a client and server via web services and for this capability to be in the entire install base. Putting that together with moving to SOA seems to indicate that you may be able to swap out and add external services right the way through core - so not only will installs be able to talk to each other but also their components. Apart from performance benefits from parallelisation and spinning tasks off to specialised apps (e.g. queuing and message orchestration, search, analytics, CRM), it opens the door to useful applications such as image servers, commerce, federated social networking, shared users.
I wouldn't say that Drupal
I didn't mean that, and yes, I agree 100% with you about a clean API.
I wouldn't try because of the poor performance of each request to the Drupal instance serving data. That's why WSCCI has been (somehow) advocating for a lighter bootstrap and (truly) overall lighter rendering pipeline.
I must agree 100% again. Seeing Drupal being integrated into bigger workflow and large-scale applications that use Drupal is appealing. Transforming Drupal to a REST-Server is THE way to enable it.
So it seems we're on the same line in the end :)
I wouldn't try because of the
I'm still understanding the proposal but not convinced on whether we are actually missing out on that much. Currently you can also use the ESI module. Although that does a full bootstrap per block, it can still equate to better performance than a single uncached full page render (depending on physical architecture, how cacheable most of the blocks are, and how heavy the dynamic parts are). Bearing in mind that Drupal's aspect-oriented nature requires us to reflect across the full stack to action any desired side effects and gather context - we can optimise it with a hook registry and so on, but I'm not clear that it can be a lightweight operation in the sense that a non-aspect request could be. For example, even a plain content read might trigger a write to something like userpoints or statistics which itself may cascade through the system. While that's a feature and not a bug, I don't yet have a clear understanding how Symfony helps. Web services gives us some very useful strategies - parallelisation and task offload, caching for reads, batching for writes, queuing for async - but these are at a bigger level of granularity than what I think is being proposed here and they are available now in D6+ (and used successfully in the wild). I think the proposal is not so much better performance out of the box but an architecture that has more points where we can tweak.
Money quote
I think that's a much better summary of the performance advantages of this approach than I've been able to come up with. :-) Switching to Symfony doesn't automagically get us a 500% performance boost. But the architectural changes that we're pushing for (whether via Symfony or not) open the door to additional work getting us a substantial performance boost. It will still be additional work, but it will be possible.
Some context (HA! See what I did there? ;))
The short version:
What would really help the "99%" of developers (as well as the 1%) to better understand where WSCCI intends to take Drupal 8 is to see some sketched out code of what a module would look like 12 months from now if everything Larry wants to see happen, happens. This would help engage people who are nervously sitting on the fence and not sure what to think. Larry's offered to give this a stab.
The long version:
We just had an extremely LONG discussion in #drupal-contribute to try and get to the bottom of chx's concerns, and some of the other concerns expressed in this thread. I'd like to provide a bit of context for others coming into this thread, particularly for those who aren't natively fluent in PHP:
There've been a lot of questions here about how these changes affect day-to-day module developers and site builders, and why some people are getting so upset about the very idea of this change. So let's break it down:
I don't think there's anyone who would argue against Drupal's bootstrap process needing a significant refactor. The biggest user-facing indicator of this is that right now, when you type the letter "a" into a taxonomy autocomplete field, you get back a simple chunk of JSON with 10 or so matching term names in it that gets displayed in the field to select. However, getting that simple chunk of JSON means Drupal performs a full bootstrap (meaning it starts up the database, user session, and all subsystems like Form API, AJAX system, plus loads up all modules, fires up a couple of hooks, performs DB queries, etc.) in order to get it. This has pretty severe performance implications, particularly as people start building more and more performance-sensitive things like mobile websites and web services on top of Drupal.
The sandbox in its current form has Zilch, Zero, Nada, No impact on existing module developers or site builders. You can see this for yourself by popping open any of the modules you know and "love" from Drupal core in Crell's sandbox, and you'll see the code is identical. hook_menu() is exactly the same, hook_block() is exactly the same, etc. Of course, this isn't the full picture, and doesn't actually gain us anything as-is; it's simply limited in scope to see what it would look like. But the actual scope of changes for day-to-day module developers is actually something I think we have quite a bit of control over, since Larry's proven that touching nothing and replacing the boot loader with Symfony will still allow Drupal to serve a page.
The code refactoring in this experiment is happening not at the module developer-facing level, but at the "get Drupal up and running" level, which is transparent to nearly all Drupal developers unless you're one of the couple of dozen of core developers out there who have the various DRUPAL_BOOTSTRAP_ phases memorized, spend your days and evenings tinkering in core's /includes directory, etc.
Some folks, like chx, ARE one of those people, and arguably THE person who's put the most consistent time and patches into Drupal core in the past 5 or so releases. When he sees sketched out code with comments like "this could be a hook replacement mechanism" his hackles start to raise because he feels that the very essence of what makes Drupal Drupal is being attacked. But let's leave the 'What is Drupal' question out of this for a bit.
However, the proposal here is just that: a proposal. It's a weekend coding exercise to demonstrate what Drupal could look like if we utilized a streamlined web framework in place of our current bootstrap process, and should be evaluated on those merits.
Now. With all that said, I think what would really help the "99%" of developers (as well as the 1%) better understand where WSCCI intends to take Drupal 8, is to see some sketched-out code of what a module would look like 12 months from now if everything Larry wants to see happen, happens. I think this would help engage people who are nervously sitting on the fence and not sure what to think. Larry's offered to give this a stab. Yay!
What initial benefits can we get without disrupting contrib?
Hi Crell. Thank you (and everyone else who's been participating in WSCCI) for the deep thinking you've been doing on this. At a high level, I agree with the desirability of:
If we have to refactor much of core/includes to accomplish the above, I'm fine with that, so long as the result is comparably performant at generating a normal full-page HTML response. To the extent that we can leverage existing PHP libraries from Symfony or other projects, I think that's great.
But I also share the concerns others have raised here about introducing more disruption to contrib module and theme developers than necessary. Some (even a lot of) disruption is part of every major release upgrade, but I think each sweeping API/architecture change that affects contrib developers should be thoroughly discussed in a corresponding d.o. issue. I get that at this stage, we're all still cloudy on these details, and I hope the concerns raised given that cloudiness are useful to the people working on this in getting the conversation/code to the next level of detail.
Is it possible to introduce some of these sweeping changes in decoupled chunks? For example, do we need to get rid of (or seriously modify) hook_menu() and hook_block_* () in order to realize the above benefits? Or is it possible to start with a patch that includes the relevant Symfony/Lithium/Pimple/whatever libraries, refactors bootstrap.inc, menu.inc, common.inc, and adds default controllers that still call hook_menu() and hook_block_* ()? Would such a patch let us improve the response time of user/autocomplete/* (a prime example in core where almost all of bootstrap is not needed but is done anyway) if we add a suitable controller for it? If so, then that can move us forward, and then let us start debating the pros/cons of forcing all modules to change from menu and block hooks to controllers. And maybe we'll decide that it makes sense for D8 to support both models, easing the transition for contrib maintainers.
I also have a concern about the dropping of hook_page_alter(), and making each block return encapsulated and immutable HTML. While I appreciate the benefits of this, please have another read at Designer error 1 from http://developmentseed.org/blog/2009/mar/01/limitations-drupal-theme-layer/, one of the things that prompted the D7 render system. We need to gather up use-cases from Drupal front-end developers of when they need to move an element from its default block to some other page location for that custom site they're building, and we need to make sure that we give these developers a way to do this that's appropriate for their level of PHP expertise, and I suspect "implement a block controller subclass for each of the blocks you're modifying, do some deep manipulation of a request/context object, and implement preprocess functions that react to that context change" will not fly. I think this problem is solvable, but I just want to make sure we're not ignoring it.
Multiple answers
Lots going on in this comment, so let me try to separate it out...
1) Originally we were trying to do a decoupled-chunk approach. That's why we spent months working on just the Context API, submitted it to core, and ran into a wall of "I don't get it without seeing how it affects everything else". Two people, including Dries, suggested "just cut to the chase and start with shot 4" (the new layout system). So, that's what this branch is doing.
2) The current wscci-symfony-dic branch is very close to being "adds the symfony stuff but doesn't change anything else". It does it in a kinda hacky way (duh), but it does it. Presumably the next step in actually implementing it for reals would be to replace the hack inside the controller resolver with a properly encapsulated mapper/resolver/thingie with an API we want, that simply sub-calls to hook_menu et al for now. Then we can mutate that bridge object into a real new implementation while building other stuff on top of it in a sane fashion.
That said, yes, to get the full benefit we are after it does require rethinking hook_menu and hook_block_* completely. If we don't, then we don't get the new page building flow that we want to get to... and that was the entire point of WSCCI in the first place, way back when it was still called Butler. It picked up web services along the way because both require considerable changes to the routing process, and it's silly to not consider both at the same time. A pull based block-centric layout is the goal, and has been the goal since DrupalCon SF. That cannot be done well without a drastic rethink of the page building process.
This touches on sun's scope comment below, so let me reiterate that: If WSCCI ends up being a wash on bootstrap performance, and the block-centric caching doesn't buy us as much as we think it will, and overall performance doesn't really budge; but it does get us a pull-based block-centric layout, native, non-hack support for non-HTML-page responses, better decoupling and testability through fewer globals, and a more consistent and predictable API through a common plugin system, then I will still consider it a success.
WSCCI's scope explicitly includes reversing the page building workflow to be more Panels-y, and always has. The only reason the initiatives name is "web services" is for marketing reasons, because Dries thought that was more marketable than "Butler".
3) hook_page_alter() is part of the problem we're trying to solve. The "the Drupal theme system doesn't let you do that" problem described in that DevSeed article is exactly the sort of problem that the block-centric layout is intended to solve. It is a symptom of the "build the page body, then throw context-free stuff around it" workflow that Drupal 7 still has at its core. Adding "oh yeah, and do it in an obtuse, proprietary, memory-hogging way so that we can allow advanced PHP developers to manipulate enormous arrays in site-specific ways" doesn't fix that problem.
Moving instead to a pull-based model, however, does. You want to put the node title outside the normal "body" area? Fine. Page title is its own "block". Move it in your layout. Done. You want to put comments on the right side of the body instead of below it? Switch to a 2 column layout and move the comment block to the right region. Done. No subclassing, no preprocess (or was it process?), no arrays that even experienced Drupal developers can't grok. Just moving stuff around in a GUI.
That's where life gets easier for designers/themers. Think Panels and Panels Everywhere, but implemented in core properly with modern PHP 5.3 techniques and a well-designed UI rather than tacked on to core as a hacky afterthought using legacy PHP 4 techniques with a UI that was simply evolved as-needed.
So yes, that problem is not only solvable, it's the entire reason this initiative started in the first place. It's just a long road to get there.
How granular are blocks?
So, right now we have hook_node_view() that lets modules add stuff to a node display (whether the node is on its own page, or in a teaser, search result, etc.). Book module adds book navigation. Comment module adds the comment thread. Blog module adds some links. Translation module adds some links. etc. Suppose on my site, when I'm viewing a node/X page, I want the blog links in the left sidebar, and the translation links in the right sidebar. Does this mean that every module needs to expose its node links as blocks and we remove hook_node_view()? Does this extend to "smaller" entities like hook_comment_view() as well? Does everything that is currently a tiny little render element become a block, because someone might want to display it in a different page location than the default? Have we just replaced an unmanageably large $page array with an unmanageably large collection of blocks? Don't get me wrong: I'm not against moving from a giant $page array to some level of pull-based block layouts. But I'm skeptical about making blocks too granular, and so I think we're still left with the problem of each block containing a render array, and needing to have an API for moving some elements from one block's render array to another one.
There is a related issue
There is a related issue here...
http://drupal.org/node/1164718
Still thinking push based
SuperBlocks/Controllers should be more granular than core blocks now, but less granular than FAPI/Render Arrays. Where exactly that line is, I'm not entirely sure. That is something we can easily experiment with, in both core and contrib, and evolve as needed.
Render arrays are far too granular. The overhead it introduces (for the developer, the themer, and the CPU and RAM) is simply too much. Supporting caching at any arbitrary HTML element nee array key is the wrong approach, and allowing minuscule editing of essentially every HTML element in array form is simply not worth the effort, nor even useful, IMO.
The guideline here, I think, should be Panels/Panels Everywhere. It's been using this model for years, and has a pretty decent set of content panes (cum WSCCI SuperBlocks cum Controllers) that work well. I believe they go down to the Field level for the most part, although most things are still rendered as a full node in practice on many pages; below that, you break out your text editor and attack a template file. Which, quite frankly, is fine, as virtually every themer I know would rather edit an HTML template than try to dissect a deep, complicated array. (There are several issues, including one I linked in the OP, where people note that in practice most themers have a hard time dealing with the contortions of render API.)
As to your concern that you need to "move some elements from one block's render array to another", you're still working on the assumption that it's only possible to get at data from the one block/region/renderable where core pushes it. That's exactly the core mistake we're trying to fix.
Rather, the pool of data is there in the system. A block is configured to suck that data out of the database or the request/context and display it. Want a different piece of data? Configure the block differently. Want a piece of data repeated 4 times? Place the same block (or a similarly configured block) 4 times. Want to suck a field out of the node view block and put it in a sidebar? Place the "field" block with appropriate configuration in the sidebar and disable it in the node view block. Done.
All of the above works with Panels today. Most (although admittedly not all) of these questions have already bee sorted out, or can easily be sorted out by mixing in Views and custom display modes without any additional code. Conceivably a Display Suite style "inner layout" for a given display mode is also possible (although I defer to the UX folks on exactly how we want to do that in practice; I'm focused on the engine to enable it).
None of these UI models is really new, even to Drupal. They're just new to core. They're not being created from whole cloth, however. In the Panels sites that Palantir has done, I think we've rarely had to implement new content panes / blocks / controllers. Usually we're implementing new ctools context plugins (what in this model woudl be additional request context) or access plugins (which are not part of the display pipeline per se, but should be addressed by the Plugins portion of WSCCI).
I totally get the concern about flexibility, but I think 1) the "absolutely infinite flexibility of everything, anywhere, any time" of render API now is not something we need or even want to retain and 2) the level of flexibility this model offers has already been demonstrated and proven "in the wild".
Scope, once again, please.
I'd highly appreciate it if people could stop making that "lighter/faster bootstrap" argument for this change proposal, because it is invalid and simply not true.
If there's anything in Drupal's bootstrap that isn't required, do you really believe it would still be there? Obviously not. Drupal still needs to initialize its subsystems (configuration, cache, database, session, language, module system, theme system, etc) and no matter how you're going to change the bootstrap process, these things still need to be initialized. Period.
As far as I can see, the effective change of this is to change/replace menu_get_item() and menu_execute_active_handler() to start setting up a $context/$request object that is then passed down to the actual callbacks for the page.
It does not touch or replace Drupal's menu router system. The only change to it is that dynamic/named router path arguments are no longer provided by procedural
%NAME_load()callback functions, but some (oddly named) "service" class implementations, registered in a "service container."In fact, the hook system, modules, and themes are not converted into OO classes, which in turn means that we will still have hooks like hook_menu_get_item_alter() and hook_init() to allow modules to perform manipulations before the request is executed. In turn, this still requires to load all modules. And we won't drop that, for very good reasons.
Furthermore, that service, controller, and event dispatcher system still needs (weighed) implementations to be fed in, in addition and likely very similar to the already existing and highly optimized module_implements cache.
In turn, we will be loading more code and need more memory to handle a request. Not less. That, combined with the necessity of having to discover and auto-load service implementations one by one, will ultimately make the bootstrap slower, not faster.
We will be able to gain some bootstrap performance improvements by converting some subsystems into classes and loading them only on demand, but again, that's completely, entirely, and totally unrelated to introducing a $context/$result or $request/$response concept in Drupal core. Neither is a Page manager/Panels/Context-style way of populating page regions relevant for this in any way. Nor is a total revamp of Drupal's module/hook system. It would be tremendously helpful to split these "sub-projects" from WSSCI into own initiatives, as I'm really getting sick of the sheer endless scope-creep and wishy-washy arguments in discussions like this. Please.
I wanted to comment on the actual prototype, but only see myself mostly looking at code, comments, class interfaces and implementations that have little to do with a context system.
Daniel F. Kudwien
netzstrategen
I might be missing something,
I might be missing something, but if my request only needs a Solr index and no DB, then I don't need to initialize my database subsystem and if all I want is to return a json list, then I probably also do not need a theme sub system initialized. So no, nothing needs to be initialized by definition, except for the sub system that handles lazy loading: aka autoloader and the tool that creates the lazy loaded services (f.e. Pimple). So if you need everything in a single request, then yes you might need more (aka the lazy loader). But the point of this all is to make it possible to split pages into logical and cacheable blocks which will likely not need everything and as a result you can often only dynamically generate few pieces if any and in a fancy setup even leave the caching to a reverse proxy.
And yes I know all of this is already possible via contrib, but not in a predictable way that lets you take any module and have it work out of the box.
Now the key concept that Symfony2 brings to the table is by making cacheable blocks sub requests, with their very own request and response is to bring this predictability, so there will be no more guess work if the module will in fact allow for block level caching or not. You know it will work. And in the same way you know it will work if you load it asynchronously via AJAX.
So in fact from all I have read from Crell, the proposed code lays the ground work for faster bootstrap, but more importantly opens the door to block level page construction (and it should be noted that block level page construction requires faster bootstrap). So if you guys determine that bootstrap isn't faster with the proposed direction, you likely will not enjoy benefits from block level page construction.
Again I am no Drupal expert, but from all I have heard about your current set of problems, I know that the approaches implemented in the Symfony2 components on the table solve these issues. So I am very confident that you will indeed reap benefits if you follow this path.
+1
Edit : sorry, I didn't see I could vote. Nice addition !
Not convinced
First off, I'm very familiar with both Symfony2 and Drupal, and I understand why Crell has become so interested in using Symfony2 components. In general, the Sf2 components are elegant and lightweight. They leave a fair amount to be desired when comparing architecture to Drupal 7. Now that Drupal is in version 7.10, it's looking like a very well-thought-out framework as well. In my opinion, once all of the Drupal 7 modules are where they need to be, Drupal will be exactly where it needs to be.
Maybe there are more creative solutions to Drupal's web service 'problem'? I can say that I've done a large project with intense ajax/json/REST responses, and had no problem with response times. However, I did use a 'hack' to improve performance. I basically echoed and died from hook_menu callbacks. This effectively stopped Drupal from doing a full load process. It performed great. I learned that I can do a complex sql query and a ton of node_load's in a single second; while node_save is not nearly as fast. Obviously, I shouldn't have to implement hacks to improve performance, which is why this initiative exists in the first place.
Here's an idea, why not just have a 'special' request parameter that invokes a different Drupal load process in index.php? Furthermore, a lightweight MVC framework (such as Lithium) could be integrated into Drupal to serve as a robust MVC-style REST server. This would essentially be an addition to what is currently Drupal 7.
I strongly feel as though Drupal 8 should be a smaller step from Drupal 7, than what this group is proposing. If some of us in the Drupal community are thinking that someday areas of Drupal will be re-written to work with Symfony2, and its components; we could start in that direction a little more slowly, and retain more compatibility with current modules. Maybe there are more creative and progressive steps that could be taken like implementing an ORM, implementing Twig (to address the 'block problem'), and implementing a separate bootstrap process for webservices.
I've always recognized Drupal for its node and content type structure in the database. Granted, this is only a single distinguishing feature. I think it'd be amazing if Drupal could dynamically create db tables and interact with them using an ORM (standard syntax). It's possible D7 modules could still work without modification; considering an ORM extends PDO.. just an idea..
No more hacks
Because that would simply be yet another hack. We've tried hacking around our current model enough. Something like Panels/Page Manager is one huge epic hack, because core's current architecture requires it. Yet that is, I would argue, a far better model for building a page than the "load world, build unnecessarily big array, modify it from obscure places to be even bigger and more memory hungry, render it in a way that no one can understand because it's sometimes rendering inside out, other times outside in, then print". Adding a "skip that mess and do something else" flag is simply a hack around a fundamentally broken pipeline.
Drupal already has an ORM: It's called EntityFieldQuery and entity_load_multiple(), which is, I'd argue, far better than a traditional overly-heavy "ORM".
For background on why Drupal doesn't use a classic ORM, see these two articles about the development of Drupal's DB layer (one written before, one written in hindsight):
http://www.garfieldtech.com/blog/abstracting-databases
http://www.garfieldtech.com/blog/orm-vs-query-builders
my 2 cents
I fully agree with the architecturial goals of WSCCI - we need to make this conversion. However, I'd prefer to an approach keeping the necessary changes at a single point to the bear minimum and so taking step-by-step. That way, the consequences, advantages and disadvantages of each step can be controlled much better. E.g. - why bother with creating a new response format? We already have a "self-contained" render arrays with attached js and css. Changing that is something that should be discussed in its own issue, until then let's stay with the usual way.
@proposal:
While I've not beeng diving too much into the code and all the previous WSCCI discussions, I really like the idea of adopting the httpkernel, re-use code and becoming more compliant with other PHP libraries. I'm not sure about using request objects as context replacement though:
When passing on a request to a rendered block, is it possible to change the "node" context to a different node? If so, we are kind of lying to the block as the node-context of the request is something else. Thus, I'd prefer the real request to be "globally" available to any code, while the block has its own context.
Related, I think we should keep the context notion, as I don't think block developers mentally think of returning request responses. You are rendering a piece of data depending on a certain context. Also see my above comment.
Then, I really like pimple's light-weight, simple and elegant use of closures. - it seems to fit to Drupal very well. I could see us making quite some use of it.
The way I'd love to see WSCCI to move forward is to take the ideas of both approaches and
* take over symfony's httpkernel and implement a RESTful approach as done by this approach
* implement a simple light-weight context system, maybe based upon pimple? Leave it more advanced stuff like stacking contexts or locking them for now and add them in afterwards if we see the need.
I think that both parts don't necessarily depend on each other and can be done in separate steps.
That's where OO is helpful
If you look at the code in the dic branch, most of the actual block/controller classes have an inner method that just returns a string or render array. The response object itself is handled by the parent class. The typical random Joe developer would still only be returning a string or render array. But if they want to take over the returning of the response object itself, they can easily do so.
Re context vs. request, my main concern is that if we use HttpKernel it requires a request object, not a context object, and the context object would have to have access to the request anyway. I'm not sure yet if we need to have HttpKernelInterface used deeper in the system than just at the top level. I need to do more research. If we don't, then we could potentially pass $request->context, since as I have it implemented the context property has a reference back to the $request anyway. I guess I just don't see the value in doing so, especially when the request object already has a "duplicate me" operation on it to allow doing exactly what we're doing.
Yes it does, and that is exactly the point. Lying to the block is the entire goal, because that's what dependency injection is. It's a way for you to control what "Truth" a given component sees, completely, so that you can use it in a different context. That's exactly what the menu system in D6/D7 allowed us to do, and it was a very good thing.
Making things global automatically breaks all the work we're trying to do in order to make code reusable, injectable, and testable. As anything but a stopgap for legacy code, it's a bad idea. The end goal long-term should be to eliminate all globals from the system entirely, without exception. That's been understood in the CS/SE world for decades. :-/
The "advanced stuff like stacking contexts or locking them" exists only to support legacy code. If we had no legacy code, we wouldn't need to do that. So we either need to rewrite everything completely, including hooks, to always take an injected request/context or else have something like the stack/lock mechanism or similar to emulate global access to request/context in a controlled and non-crappy way.
At this point I'm trying to be relatively modest and not propose we rearchitect and rewrite rewrite every line of code in Drupal all at once. Just a large portion of them. If everyone else thinks we should really go all out and rewrite everything, well, I'll go along with it but I'll be very surprised.
context vs. request
Still, the request actually happening is "global" to the code running. Yes, we can and should encapsulate access to it via the httpkernel to get these advantages, but I don't see the value in cheating on other, regular code to believe that another request happened. I do think that this kind of obfuscation makes it in the end harder and more difficult to develop as it's more difficult to get the big picture of what's going on.
Thus, if I write a block dependent on a node context, I should receive the appropriate context, i.e. a node. And I should be able to easily access the actual request object, whether it was a full-page request with a totally different node context or a request to handle the block only. Yes, doing so might have consequences on caching and so on - but that should be up to the developer, so he still has full control over what's happening.
No, the menu system does not lie to me. It kind of builds the context for the page and passes it on. We need to derive context for the block too though and we need to know what context the block needs. Once we know that and have control over it, we can easily re-use the block elsewhere without having to lie. We just tell the block "that's your context, go with it". It might be not much different in code, but to me the mental model is quite different.
I see the value in improved DX. The objects around my code should represent what's happening, not emulate a not existing situation.
I'd only go and mock request objects for testing and maybe as last resort, but not as part of a regular API.
If it's just for legacy stuff, so why not leave it out for now and implement the system as it should work? Once we are going this way and see we need legacy support, we could still add that.
No, I am not sure if you are
No, I am not sure if you are unclear about the proposal or if you just agree with the goals, but again the entire point of this all if that a sub request is a standard request. If there is any global state involved in creating the response there is no way in hell you can reliably and predictably guarantee that a given sub request could also become a "master request" which is what is necessary if you want to have the option of delegating caching to a reverse proxy or lazy load content blocks via ajax.
Every sub request must be handled via a new request instance that gets passed to it all parameters that are relevant explicitly rather than letting it access what it decides to need from some kind of global state.
I got that, I'm not arguing
I got that, I'm not arguing to introduce global state.
If every block is rendered upon a known context, we can create the context either from a parent context (full-page load) or directly from the incoming request (block only). We don't have to fake request objects to reach that goal.
#NoMoreGlobals
It's global now, because when you output a piece of content in a hook_menu implementation, it's considered the main content (stored in the
$contentvariable in the theme layer), and then the theme layer brings/pushes blocks around this main content, trying to be "context-aware" with the "Visibility settings". WSCCI is trying to fix this broken model, and it's broken because :1. The "Visibility settings" are not content-aware at all, we have to write ugly PHP code to bring context from globals instead of using directly accessible context. This is what Panels is "fixing" (in a clever but hacky way because it precisiely circumvents this very problem) with its context system that you can pass to each block/panel pane.
2. The push is happening in the theme layer ! Its means that you can only get HTML versions of blocks, because you don't have any choice !
3. You can't get retrieve a block in a hook_menu implementation the way Drupal is designed, only a main piece of content (like a node, a view, an entity etc.). So you have to hack your way and get hands dirty and write useless PHP code to output a block. At the system level, it means that Drupal somehow restricts access to some of your content, because they're not treated as pieces of content, but "decorators" of content displayed in a hard to predict and inflexible way.
With a non global context, it makes block displaying predictable and controllable way. It makes them accessible like any other piece of content in your Drupal site. And, I know I'm going to repeat something that Crell already said many times, but manipulating the context and lie to blocks and others stuff when necessary when passing the context object around is Dependency Injection. This allows for testing through easy mocking of the Context object, and dependency injection is just the right way of designing applications. It makes each building block of an application independent of the others, so it's easily replicable and portable. Oh, and cacheable.
terminology
I'm all in for that proposal, what I'm caring of is the terminology. I don't see much sense in injecting request objects of requests that never happened. Injecting context sound just fine to me.
With
I meant that there is just one http request by page load - by reality - and we cannot change that. If we write code that mimics something else, we are just creating illusions developers have to get first.
You just argued against function parameters
I can take a properly written page callback today and put it at a new path without breaking anything. The author of the page callback didn't consider that it would be used at foo/bar/$nid/baz rather than node/$nid, but because he doesn't look at globals and only pays attention to the $node that's passed in, I can do that. I could even call that page callback from another page callback, just passing in a $node object that I created myself that doesn't even exist in the database. Is that lying to the page callback? Semantic debate, but arguably yes. The point is that the callback doesn't know, or care; all it knows is that it thinks its working on a node object. Period.
If, however, the page callback calls arg() to figure out its node, or the current path, or the current user, or anything else, then I can do none of those things. Globals break shit. Period.
That's not fancy OO. That's using function parameters as designed, circa 1970. Saying "but module developers should be able to bypass the entire Drupal architecture and access global data" is the same as "but module developers should be able to break every bit of automation we have in the system so we can never rely on anything." If part of our goal is to keep Drupal approachable for casual developers, then NOT encouraging them to shoot themselves in the foot and break the architecture is a hard requirement.
We can have sane block-level caching that "just works" for the 99.9% use case, or we can support arg() (and similar style behavior). We cannot do both. To me, there is simply no debate to be had there.
?
I get that and I'm totally with you. I'm not advocating for using globals. I'm arguing for writing code that reflects what actually happens, what means there should be only one request object per http-request - not one per block or so.
Ok I got you. You're saying
Ok I got you. You're saying that they're should be only one request object per-http request, and one response. Internally, each block should receive a context object, derived from the request. I understand the mental switch you want to avoid by "writing code that reflects what actually happens", so it's easier to understand the concept and write code that matches it.
The thing is that having a global request for the http-request, and then sub-request for blocks, means that at the "page" or "global" level and at the "block" level, the logic is the same. Request->Process->Response (simplified). So this way also make it easier to write code, because whatever response you're creating works the same way. It's just that the logic doesn't match the actual page building process.
Keep going
Short version: Lets keep going with this proof of concept. I can't tell yet if this is a net win for Drupal.
In order to evaluate this patch, I fired up my debugger and requested node/1/demo. I can't stress enough the usefulness of getting a debugger setup for Drupal development, and for major patch evaluation like this. I've had success with 4 xebug clients: Komodo IDE, Eclipse, PHPStorm, and Netbeans. They can be a bit painful to setup but very much worth it. Its a rite of passage :)
I think Catch/Sun/Crell have acknowledged that we don't expect an improvement from this patch in performance nor memory usage (as compared to our plan to use more autoloaded classes independant of WSCCI). So, that leaves "better render pipeline" and "better testability" as motivations. Those sound good, but I don't think we've demonstrated either very convincingly.
The render pipeline for node/1/demo looks identical to me to core with block module disabled. The router passes in the node as a param and then node_view returns a render array. Then the page gets rendered. We're going to need a much fuller vision of how regions and themeing play in here in order to evaluate if this patch is actually progress.
The testability improvement of of providing a Request object to each 'page callback' does seem like an improvement.
Symfony is a lot slimmer than what I had imagined/feared. I find the code flow reasonably easy to follow.
Symfony is behind Drupal in terms of code documentation. Same for Pimple. Drupal 7 sets a very high bar here, IMO. It is a shame to incorporate all this code which does not meet our quality standard. I think this is a non-trivial risk.
As a general rule, I think Drupal has to start paying a bit more attention to backward compatibility. We're a ten year old project now, and the risk/reward for minor improvements that break BC has turned upside down, IMO.
Having said that, I am not compelled by the argument that we should not proceed with this patch because it is unfamiliar to existing Drupal devs. It is just fine if some devs leave the project and others come in. This is a sign of health, IMO. Mixing up the people and the code every so often keeps things fun for core development. When core development becomes unfun, Drupal is in serious trouble.
Thanks a lot for writing this patch. Planning has its place, but I think it works better and feels better when we have code we can discuss. Also thanks for being bold. Bold is good.
Thanks also to the Symfony and Lithium devs participating here. Your input is much appreciated.
Responses
Debuggers: Abso-frickin-loutely. :-) A real time debugger is the greatest thing since sliced bread. Maybe even better than sliced bread.
Documentation: Yes, aside from intrinsically hard to document mega-arrays Drupal sets a really high bar for API documentation. If we do adopt more Symfony components, I'd actually suggests that improved documentation could be something we contribute back upstream. That would help future Drupal devs as well as future Symfony devs. (I don't know if Fabien would appreciate our level of OCD documentation, but I recommend we overload him with it anyway. :-) )
With regards to backward compatibility, I (somewhat ironically) just posted on that subject yesterday. Short version, Drupal is currently architecturally incapable of serious BC. The rather deep changes that we're discussing here, though, would make such BC more viable in future versions, but that's something we'd need to plan for. BC has to be a feature in advance, not in hindsight.
Also, most major projects are, at this point, seeing major overhauls. Symfony, Zend, Midgard, PHPBB, and others have all been conducting major structural changes of late to take full advantage of PHP 5.3. Now, "everyone else is doing it so we should" is not all that convincing an argument on its own, I agree, but the trends in the PHP world are toward more loosely coupled, full-HTTP-leveraging, namespace-component-using systems via heavy refactoring. So we wouldn't be going out on a limb to do the same; honestly I think the bigger risk is in getting stuck in a last-generation model.
Render pipeline: The main difference is that it does away with the "main body" and "context-free stuff wrapped around it" split. There are simply request/context-aware blocks layed out together. In the current example the node and user objects being displayed are completely equal. I would actually suggest that if we don't have hook_page_alter anymore than we can have more blocks return strings, not render arrays, which would be good all around, IMO. Think Panels, which is the conceptual model we've been working from. The rendering of one specific node doesn't change all that much, necessarily, aside from moving code around. The page-level, however, changes dramatically.
I'll be working on some additional examples per webchick's comments above. Stay tuned.
Short version, Drupal is
Just to chime in and say I couldn't agree more with this. A clear dependency chain along with using interfaces much more will give us considerably more space to make changes down the line without breaking BC. This isn't only about not breaking bc in future releases, but also the ease with which bug fixes and other changes can be backported to previous versions (i.e. from Drupal 9 to Drupal 8). And it is worth doing a heavy refactoring now to be able to make those changes easier in the future.
However this really has very little to do with changing the page rendering structure and web services when it comes down to it (or at least those are minor compared to the overall impact of such a change), and I'd really like to see that argument made in its own right rather than snuck in via these things.
Granted, the direction of the
Granted, the direction of the page flow is irrelevant to BC. I didn't mean to imply that WSCCI or Symfony were a prerequisite for better future BC. However, the heavy interface-centric approach here is the sort of approach that would, as a side benefit, help with future BC. Other changes would have to be made elsewhere as well for it to be significant. (The OOP entity refactoring, for instance, is a huge win in that regard.)
Dev tools
It would be nice to be able to use these. In Drupal, we have the awesome devel module, but using these would be helpful in addition to that.
http://www.slideshare.net/old_sound/debugging-and-profiling-symfony-apps
Software Engineer @ The Nerdery
It should be noted that this
It should be noted that this presentation is based on symfony 1.x. The tool chain inside Symfony2 however has matured even more. I think most of the relevant code is inside https://github.com/symfony/symfony/tree/master/src/Symfony/Bundle/WebPro... .. it uses various Symfony2 components and Symfony2 infrastructure of course. I don't know the code very well though and I am not sure if we could move some more code out of the Bundle into a lib. But it should definitely serve as inspiration.
Here is some more infos:
http://symfony.com/doc/2.0/book/internals.html#profiler
With a quick search I found which shows some screenshots:
http://www.decentmind.com/2010/08/symfony2-webprofiler/
There are a couple bundles that add further improvements:
https://github.com/Elao/WebProfilerExtraBundle
https://github.com/schmittjoh/JMSdebuggingBundle
One of the big features in 2.1 is a timeline that shows an overview of the request and all events that provides very context aware data that cannot be generated with tools like xdebug or xhprof. It also helps a lot with debugging event based applications.
Those are mostly 3rd party
Those are mostly 3rd party tools which can just as easily integrate with Drupal today. Lets be precise when reviewing this patch and not enumerate cool stuff that is independant of it. Drupal has syslog module for centralized logging, timer API, XHProf integration, ...
Indeed Symfony2 components
Indeed Symfony2 components totally lack documentation. I will bring this topic up with Fabien once more so that we can figure out a game plan for providing documentation.
Symfony components docs
I (and I think all of us who contribute to Symfony2) would gladly take on your OCD level of documentation :).
The main reason for the lack fo components documentation at this time was to serve our most core audience (i.e. the framework users) first. Now that that's basically done, the focus has turned to documenting the components. So, it's definitely not an oversight, just an issue of timing. Fabien is heading up the component documentation right now and I imagine once a first draft of things is posted, the community will push a lot into it.
Awesome!
It's not just out-of-band docs, though. Drupal's website docs are somewhat hit or miss at times, too, although they've been getting better. It's the inline docblocks where I think Drupal really kicks butt as far as OSS projects go. Eg, documenting what Request::$attributes was intended for (random extra stuff) would have saved us a great deal of time implementing a random-extra-stuff wrapper around the request. (OK, not entirely, but it would have saved a lot.)
That said, out of band docs on the web site would be good, too. :-)
Was reading through some
Was reading through some notes I had and ran across these:
It seems from a high-level to be Symfony2 integration in user-space with no alteration to core - this may prove to be quite useful in core adoption.
Just FYI: The goal of this is
Just FYI: The goal of this is to integrate Symfony2 and D7. The lead developer is one of the more well known Symfony2 contributors (he is behind several other bundles that are very popular).
Wrong href in links.
Wrong href in links.
Fixed
I've corrected the markup.
Django+++
The more Drupal can be like Django, the better. The Django developer experience is 100x friendlier, and the project itself suffers a lot less from the NIH syndrome.
A question
There are a lot of people who are resistant to the large-scale changes being discussed here, in other threads, in IRC, etc. Many of the arguments I've heard have been things like "This isn't Drupal anymore" or "It is too big a change to make in one release cycle" or "We will alienate module developers if we do something this big". I'm curious if these people are against the specifics of the proposals, or if they are just against any change of this scale regardless of what it looks like.
If the answer if generally the latter, then this is a lot bigger than just a WSCCI issue, and no change of the type Larry envisions will ever be generally accepted regardless of what it looks like. This is, I think, a really important point to clarify.
Different people, different opinions
There always will be people that are against changing of things, but the funny part is the same people often want more features, more bug fixes and so on.
For me the changes are needed, what is the point of new version of something if it is same old :) And as far as i am Drupal developer i prefer to see them happen, because will make the things better for everyone.
If someone don't like the changes always can stay with older version of Drupal as now there are a lot of sites on Drupal 6 and even Drupal 5.
About the module developers. With better and easier APIs will be easier for all imo.
I am working over an year on drupal 6 and now 7 sites and can't find it so easy to make things that i want to keep the old :) I understand that most people don't like learning new things and stress them self but i think Programmers and software must always evolve.
There is one other thing to consider about module developers. Many good programmers stay out of the community because they believe the things now are not they must be.
Both Panels and Views are needed Monsters, but still monsters. They are used in 90% of the sites may be even more, but they not use Drupal way or at least as i see it :) So the use case show that the Drupal Core must change so no need to hack it on daily basis to get your job done :)
Fear
I think some people are afraid of the change because it may break their DX/UX (in the broad sense) with Drupal. They may have gotten too intimate with the API and the actual Drupal way, or they may feel the change will be too big for them to adapt because of the technical changes (for instance a bigger switch to OO, using the interface differently, the new REST approach, etc.).
For these people : the guys working on WSCCI, or other large-scale changes from initiative like yours (CMI), know VERY well these problems, and will do anything they can to ease the transition. But first and foremost, at least regarding WSCCI, it's just a proposal and a proof of concept. Nothing has been committed because nothing can be committed as is : it's far from being an acceptable version for integration into Core. Leave your fears aside, and try to comment the patch for its pros and cons. If you can't, ask for explanations. When the time will come to show how-to's and precise documentation, believe there will be how-to's and precise documentation.
Count me in
When it comes to how-to's and documentation, count me in.
+1
I hope your comment will reassure people fearing the change.
Risk vs Reward
With the developers I've talked to, none are fundamentally opposed to change of any type. The common concern I hear is, "With this change and the natural consequences that result from this change, I'll have to relearn all of Drupal. I don't want to do that again."
Now, in this thread, that concern has been somewhat addressed with assurance that if this change were to be put into place we'd still have an api that is similar enough to what we currently have so that we'd have a shallow learning curve. And so far, we're only talking about using parts of Symfony to solve specific issues that the WSCCI initiative is trying to solve instead of completely rewriting Drupal on top of Symfony. So I think a lot of that initial concern has been addressed by posts in IRC, this thread and previous threads.
What remains is the uncertainty that in the end the risk of this initiative may be far greater than the rewards.
In my view, the carrot of possibly having backwards compatibility in the future alone is worth all of the risks. And the prospect of making the Drupal code easier to understand for my non-Drupal coworkers is a huge win. So while this process might be painful in somewhat-near future, the rewards seem to far outpace the risks.
I think we need to be mindful of the fact that we have yet to see the full impact of this change. As such aren't able to truly judge if this whole effort is in fact going to be for the better. We believe it will be for the better because we trust our community process and our ability to resolve issues.
Software Engineer @ The Nerdery
And so far, we're only
This is correct, currently afaik the main components under discussion are HttpFoundation (already in core but not used yet), ClassLoader (in core and likely to be used within a couple of weeks) and HttpKernel. Pimple is by Fabien Potencier the Symfony lead, but not a Symfony component.
This is not correct, the end result of this proposal if we adopt it and carry it out properly, means significantly refactoring more than 90% of Drupal code, although not all at once, and possibly not all during the Drupal 8 release cycle. But it will mean some level of refactoring and API changes at nearly all levels of the code base, both core and contrib, even if that doesn't get fully completed until Drupal 9 or 10. Larry has been quite clear about this IMO, and it's a shame that some others have tried to underplay it (not accusing you of this at all but there's clearly a gap somewhere). That's not new. There are still remnants of pre-FAPI code in Drupal 8 in dark corners of core where people rarely bother to look (like user module and comment module), despite core undergoing a major rewrite in 4.7 once it was introduced. Similarly we only have about 1/20th of an entity API in Drupal 7 core, but still a lot of contrib is being rewritten as if it was complete (commerce, rules, og etc.)
If we adopt a $request/$response paradigm, then all code below the $request (and $request in the proof of concept is initialized in the first few lines of code in what is currently index.php so that means nearly everything) that needs to access any part of that information has it as an explicit dependency. An explicit dependency should ideally be injected (i.e. passed to a constructor or as a function argument, not called out to from within a function).
Some areas of Drupal have been slowly moving towards this:
<?php
// D5 menu or block callback.
function foo() {
if (arg(0) == 'node' && is_numeric(arg(1)) {
$node = node_load(arg(1));
}
}
// Drupal 6+ menu callback.
function foo($node) {
}
// Drupal 6+ block callback.
function foo() {
$node = menu_get_object($node);
}
?>
Except that many block callbacks will check things like user_access() or variable_get() that rely on globals, so what we currently have is not remotely dependency injection at all. To avoid having to rewrite all Drupal code all at once, the current context API patch has a shiv to allow procedural code to access the $context/$request object from within functions (similar to menu_get_object() but responsible for much more), without it simply being a big global/singleton (although for the purposes of those functions they can mostly treat it as one unless they need to write back to it).
Whether this is Pimple, Symfony's dependency injection container, something we write ourselves (like the butler/context object), something from another framework again, then to use it 'properly', it's going to mean refactoring a lot of code, otherwise it's not worth doing.
I think this is another potential misconception here. Near the beginning of this process, a lot of research was done into several different frameworks, in the end Symfony2 was chosen because it is easy to use it without having to write 'on top of it'.
When talking about Symfony2 there are really two things:
Symfony2 components - this is HttpKernel, HttpFoundation and other things that are being discussed for use by WSCCI. They are self contained PSR-0 compatible collections of classes, which are roughly equivalent to what we'd call a core subsystem now (i.e. caching, logging, queueing). HttpFoundation and HttpKernel happen to be components where Drupal currently does not have a more or less self-contained/cohesive subsystem dealing with those issues - a lot of the logic is directly in bootstrap.inc/path.inc or scattered around elsewhere for example, which is why Crell is suggesting adopting them rather than writing our own.
The Symfony 2 full stack framework. This is a framework written based on the Symfony2 components (but also I think some external libraries like monolog that are maintained separately, as well as it's own code for connecting all these together). No one has suggested (or not people actually working on core anyway) rewriting Drupal to be based on this. But with Symfony 1 and many other frameworks, that would have been the only choice.
Fabien has helpfully written a blog post trying to clarify this (so it's clearly a common misconception, not one limited to this discussion) here, http://fabien.potencier.org/article/49/what-is-symfony2, which also nicely lays out his plans for world domination ;)
However this doesn't mean it's a small change. If we want to freely make use of Symfony components (or other modern standalone components that follow the PSR-0 standard, or have other projects use Drupal components instead of writing their own), then the rest of Drupal needs to follow that standard as much as possible. That means converting low level and higher level APIs to closely resemble what you'd expect to find in a modern PHP 5.3 framework (classes, PSR-0, dependency injection, composition). IMO, we should be doing that anyway if we don't want to get stuck lugging around architecture that has not been necessary since 2009 or so (which happens to be when Drupal 7 went into code freeze with a requirement to support PHP 5.2 established long before then). So really, refactoring those APIs along these lines is orthogonal to WSCCI, a new block system, or the specific proposal to use HttpKernel here.
Where it comes back to WSCCI (vs. one-by-one changes like rewriting the cache system), is that by starting right at the beginning of bootstrap and taking responsibility for the request, it implies doing this across everything (currently we are mostly picking and choosing low hanging fruit in terms of the core patches being actively worked on and committed at the moment that are relevant to this discussion). That's really about adopting a $request/$response paradigm as opposed to Symfony specifically.
So, what's really under discussion here (but is unfortunately not being discussed directly most of the time) is really turning Drupal into a framework/application that is in the same rough class as the Symfony2 full stack framework except configured very differently, and with the site building application that we know and love (and in the case of blocks hopefully hate less ;) on top of it.
So it does not look like this:
Symfony2 -> Drupal.
But instead looks like this:
PSR standards -> PSR-0 compatible components (mostly from Symfony2) -> Drupal
PSR standards -> PSR-0 compatible components (mostly from Symfony2) -> Symfony2 full stack framework
PSR standards -> PSR-0 compatible components (mostly from Symfony2) -> Silex micro-framework (and other frameworks, and applications such as PHPBB3 that go this route).
There are a very large number of unknowns with how this might look. The cache and lock changes that are in progress in Drupal 8 to make them match this model are similar to what happened with the db layer between Drupal 6 and 7 (i.e. API changes and massive internal refactoring, but not much in the way of major conceptual changes for developers in the sense that field API was). They are API changes, but it is mostly for people implementing backends, or find/replace.
No-one really knows how this might end up looking for hook implementations, or something like FAPI which does not lend itself well to classes and interfaces for various reasons. Hooks and FAPI are really the two main APIs that are particularly alien to this model, that could be very hard to move over to it.
Drupal concepts that both site builders and developers interact with and learn (nodes, blocks, views, fields etc.) aren't really affected by this at all yet, even if the actual code might change. Symfony and other frameworks do not really tackle those problems at all. There are some other projects (PHPCR, Doctrine) that do to greater and lesser extents, but which aren't under serious discussion at the moment and adopting some of this does not imply pulling those in too (although they would be less alien to integrate with than now).
There are risks both ways.
there is a lot of work involved if we want to go this route, so it needs a lot of people on board to carry it out, and a lot of the work has very little to do with the explicit goals/features of WSCCI (more the implicit ones). At the moment there is disagreement both on the overall goals (core more closely resembling a PHP 5.3 framework) and how necessary it is to achieve the specific features that WSCCI is concerned with (web services and pull based block layout). Unfortunately much of the discussion tends to dance around this a bit, but also people are focusing on specific API and implementation details and extrapolating outwards from those, which also needs to happen but can be hard to follow in terms of the overal meta-discussion about direction.
even with a lot of people working on it, it's the sort of change which could determine the length of the release cycle rather than being determined by it (there's only so much we can do half way with this sort of change). Drupal 7 just came out from an unexpectedly long and unpleasant (especially the final 18 months to 2 years) and a lot of the people who worked on that are still feeling a bit weary from that experience. Personally I think this would require a long release cycle, but it would be differently long hopefully.
that Drupal will lose its uniqueness and become indistinguishable from other PHP 5.3 frameworks (which currently are not evern discussed as 'competitors' when say Dries is talking about Wordpress vs. propietary CMS. IMO most of the things that would be affected by this (not all), are not really that Drupally. You can go quite a long time without understanding bootstrap or the cache API. The things that make Drupal unique the fact you can configure so much raw application logic using UIs, concepts that span APIs and interfaces like nodes, taxonomy, fields, entities, views, rules, and some of our architectural uniqueness like hook_form_alter() and hooks in general. There are going to be places where some of these things clash a bit, but we already have that now IMO.
on the other hand, if we don't start making these changes now, we'll have an architecture based on pre-2009 PHP, and will need to support that until 2017 or later. And if we ended up doing this in Drupal 9, that might not come out until 2015, so would be six years behind current trends in the rest of the PHP world instead of 2 or 3.
Using interfaces and trying to separate concerns as much as possible makes backwards compatibility easier, it doesn't mean that major core versions won't break it (but they might break it less, or differently). However that is something that's only possible after doing all this work, you don't get it while the work's being done. One exception is that it's quite easy to backport some of the refactored APIs to Drupal 7 or even Drupal 6 so module developers can start using them now. http://drupal.org/project/dbtng http://drupal.org/project/cache_backport and http://drupal.org/project/drupal_queue are all good examples, all of them are using interfaces.
It will be easier for people coming from other PHP projects that are based on PHP 5.3.
There are a lot of concerns being expressed here and elsewhere as to whether it will be harder for people new to programming (many, many Drupal developers, including me personally, started out as 'site builders' configuring the CMS, then worked their way into development once they hit bugs or limitations that required writing PHP).
For those 'new developers' and sometimes 'accidental developers', there are two parts of the learning curve. Can they learn enough to do what they specifically want to do quickly? And once they know that, can they then apply it to the next thing they need to do? Drupal scores very well and very poorly in this regard depending on what it is specifically you're trying to do at the moment. How this changes it for 'new developers' as opposed to 'developers new to Drupal' I'm not sure yet, it is not particularly easy remembering what was easy or hard in 2005 (I remember I found Drupal 'concepts' like nodes and taxonomy very natural and easy, but actual code like wtf a form #submit handler is and how it gets called bizarre), and I can't unlearn Drupal now, go off and learn a PHP 5.3 framework, then come back and look at it fresh from that angle either.
A significant part of the existing contributor base to Drupal, didn't come to it from other programming languages or PHP projects, but came to it from "I need a build a site, oh look I'll use this, two years later I'm a Drupal developer". The current Drupal learning curve is not easy, yet many people are learning it, and those people will be quite capable of adapting to PHP 5.3-era concepts instead of PHP4 (in the same way many people already adapted to automated testing in the past three years compared to the entire project mostly getting by without it for the previous 7). But how this impacts people who open up a Drupal file and see their first ever PHP code is obviously going to be close to the hearts of people who did that 4, 5, 6 or 9 years ago. I personally think that aspect of it should be fine - a lot changed elsewhere in the past 5-6 years, and a lot more people are using Drupal after already having used something else (how many migrations to Drupal were there from other CMS happening when 4.5 came out).
Note at this point I've not reviewed the branch in Crell's sandbox, nor had a look at HttpKernel yet. I think that work is necessary to provoke this discussion, but this is really about project direction more than either APIs or implementation - so we should be explicit about that and discuss it on those terms. That may well require fleshing out the proof of concepts more so people can see real code, it also means being brutally honest about what's actually being discussed here.
Thank you. This is the most
Thank you.
This is the most accessible overview of the challenges we are facing, and I believe it is important to distinguish them from the actual patch being discussed.
This is what we need to form consensus on, and commit to it.
Then the actual implementation is a matter of our usual reviewing and nit-picking, multiple iterations and approaches. But without consensus on the core points above, we are not going anywhere, because it will always stop on "this is too much, this is not Drupal (in its 2006 glory)". Yes, I am concerned about introducing a ton of complexity, but that's why I want to participate, review changes, and try to help out. We introduced a ton of complexity with Drupal 7, so it's not a problem that is unique to WSCCI, Crell, Symfony, or whatever.
Complexity
The point about complexity is important. As I understand WSCCI, whatever content is displayed in the page will be a (super)block, no more division between the main content and a block put around. They will both respond to a request/context (TBD). So whatever the final API the fact that it's same approach for every piece of content makes it simpler by definition to... output content. WRT to displaying content, I think WSCCI will make it simpler.
With all the refactoring needed, I deeply believe that the OO approach will standardize the way developers interact with Drupal "stuff" : entities, configuration, cache, database, forms (tricky, as catch said above), etc. Some of these parts will use the same patters for interaction, CRUD and so on. We'll have interfaces, and implementations. This will make stuff easier.
And every pluggable stuff in Drupal will follow the same pattern of plugability, instead of being spread between different one-off implementations (some in settings.php, some in pseudo hooks, some relying on Ctools etc.). This will also make stuff easier.
Of course, nothing has been done yet, but I think from an "API consumer" point of view (as I am, and as many in the community are), this will benefit the DX, because even if we have some new things to learn, we won't learn that many new things, so it could easily balance the (yet to be seen) complexity brought by WSCCI (and other initiatives as well).
Last thing : as catch and Crell said rightly, the patterns we want to bring in (request/response model, OO, PHP 5.3 possibilities etc.) are also being used outside Drupal in the PHP world, and if you take out PHP specific stuff, outside PHP as well (Ruby, Python, Java, Javascript...). Many developers in the Drupal community that also used others technologies could say : Drupal is very specific in its design. So being "more conventional", not for the sake of being conventional, but for using well known and proven patterns, make it easier for Drupal people to use other technologies, and more importantly, bring in more people in the Drupal game. Remember we clearly lack experienced devs in the market. Remember also that when Drupal 7 was released, several people left Drupal because of it was being too complex. WSCCI (and others initiatives as well) is a chance to push the API clean-up even further.
If it wasn't clear, I'm conscious about the risks, and I'm all for what WSCCI wants to bring to Drupal.
Thank you catch
I tried making a similar list of risks / rewards earlier but I lost that post (must of clicked preview instead). You did a much better job of explaining those risks and rewards on both side of this issue.
What you've said here matches my understanding but seems to contradict what webchick was saying earlier about this was really not that big of a change, thusfar.
Personally, I'm all for this kind of Drupal adventure. I've wanted to refactor Drupal as a intellectual exercise a few times before and now it seems that we'll have a lot of people to look at that.
Where I work, my biggest challenge is getting developers that are very gifted in writing custom PHP applications (Zend mostly, or our custom use of it. A few Symphony folks) interested in working with Drupal. Presently, there is shortage of specialists who can develop custom modules for project (you may have heard of this before) and there is a considerable ramp up for developers to get comfortable with the Drupal API and Drupal way of doing things.
Ultimately, I think this transition will be an inflection point for Drupal development due to the familiarity that professional PHP developers will start off with. If they are comfortable coding Zend or Symfony they will at least understand a decent percentage of the code Drupal uses to be awesome.
Catch, Crell, Webchick, or any other Drupal core developer, do any of your travel to give Tech talks? I'd love to have you guys provide a perspective to all of our PHP developers here. This sounds like an awesome topic.
edit: spelling fixes
Software Engineer @ The Nerdery
Minor point
Symfony2 was chosen for the context patch to parse out and store the HTTP request headers. I have yet to see the research for the general case.
Perl 6?
Catch's post reminds me of a historical parallel in another software community -- the Perl community, of which I'm also a part. Perl 6 was designed as a complete rewrite (or 90% rewrite) of the Perl code base beginning in the year 2000 and still continuing development today (yes, 12 years ago -- thank goodness they forked and kept up with Perl 5 release cycles!). While we're dealing with a framework rather than a language, and major vs minor version upgrades, it's still illustrative of how and why the community has essentially forked Perl 6 as a completely separate initiative, and continued to dedicate the majority of its energy towards minor version upgrades of Perl 5, backporting certain essential features of Perl 6. Perl 5 is still absolutely awesome, even though it doesn't have some of the lovely architectural features of some of the other dynamic languages, such as true object orientation (although it accomplishes sweet OO via -- you guessed it -- a module). What's more, they have www.cpan.org -- an absolutely brilliant, enormous, fruitful playground of Perl 5 open source modules, supported by a smaller community but having a longer module longevity due to smaller architectural increments in Perl 5 that are mostly easy to port to newer versions. While Drupal is different in terms of breaking APIs and porting between major versions, we still have a good deal of basic consistency and things like the Coder project that at least somewhat helps port modules from D6 to D7. The Perl community also understands that they'd be nothing without CPAN -- the large base of contrib modules created and maintained by the community -- as one of its greatest strengths, and they understand that Perl 6 is nothing without that.
We developers tend to focus on purity, the "best" solutions, architectural bliss, and we really want to upgrade to Perl 6 even though it doesn't make sense at all in any context other than hacking in our basements. However, if we don't think about this in a cost-benefit paradigm, and understand & evaluate the benefits of getting things like the D8 config management initiative out the door, given a fixed amount of resources for all of D8, versus having a Perl 6-ish architectural heaven of Drupal RESTful core, then we might be steering in the wrong direction. Of course, the D8 config management initiative will require some core changes, but from what I'm understanding it's not on par with the level of change proposed here.
After spending a year as CTO of a Drupal social network site and feeling the pure pain of dealing with Features, config management, and releases in Drupal, I can't see how something like config management wouldn't take a bit of precedence over many aspects of WSCCI. And now, seeing Estee Lauder (where I've been hired as The Drupal Expert) evaluate Drupal as a technology of choice for its 27 international brands, and its engineering team totally digging most aspects of Drupal and able to learn it but wondering why they have to jigger up some jankity Features or write install/update hooks just to deploy changes to Very Important Sites that cannot break or experience downtime because of some weird deployment nuance, I really see some real-world priorities that would seem to outweight some changes that might make our lives as developers a bit more easy and streamlined.
For now, we do have the Services module, and we can write hacks if we really want to use Drupal for web service functionality versus using another framework or language. I just wrote something the other day to geocode some stuff and spit out XML -- great. Maybe there's even an in-between solution that wouldn't be perfect but passable until pure Perl-6-style architectural nirvana can be achieved (i.e. developing a module and keeping this particular stuff out of core...people were talking about doing something with the Context module, right?). It might even be ok to admit that the developers who really want to do all the crazy stuff proposed by the WSCCI are of the skill level to do it in...another framework, or another language, or via RPC calls to a secondary service, etc.
Drupal specializes in making really awesome web sites really quickly. My humble opinion is that it's important to evaluate, as this discussion continues, exactly how complex things are getting, how far we're straying from our specialty, and how closely such decisions towards complexity parallel lessons from history in other open source communities. If this isn't really that big of a deal, then I'd say go for it -- but I'm guessing catch knows what he's talking about ;)
sethviebrock.com
CMI is quite a small API to
CMI is quite a small API to add, but it is going to mean serious refactoring of every core and contrib module to use it properly as well as providing a mechanism to upgrade configuration from the old one-off systems and the travesty that is variable_set(). If they don't get refactored to use it, you'll be stuck with features in Drupal 8 (and I really, really dislike features and hope CMI kills off as much of it as possible). It is not 'some core changes', it is 'completely replace the storage of everything that's not content'.
However the architectural considerations of CMI, and the related work of things like completing the entity API, are part of exactly the same process as this conversation (in case you missed it, there were just as heated conversations about CMI only a few months ago, although mainly focused on details rather than the overall goals).
When you talk about 'real world considerations', that's a euphemism for 'feature requests for my business'. Drupal core is still 99% a volunteer driven project (even if many people earn a living doing Drupal, a lot of core time is completely donated), and people do not react well to feature requests when they're not backed up with offers to help. So I hope to see you working on some of the CMI conversions you're so looking forward to when the API lands.
Drupal core development is also the real world, in the very real sense that hundreds of people are putting thousands of hours, usually of their own limited free time, into something that Estee Lauder (or my Mum) get to use. Cramming in features on top of an old architecture meant that Drupal 7 took about 18 months longer than it should have to be released, leading to serious attrition and burnout of the people who helped get that release out the door, including me. Keeping going mostly the same way while adding a few features that people are clamouring for does not appeal to me after that.
I Apology if i understand your wrong
If i get something wrong from your statement please excuse me and correct me.
As far as i understand your parallel Drupal 7 Architecture is the "usable" ( making sites and money) solution and this initiative for Drupal 8 is just for programmers to fill happy playing with new toys.
I don't think you're write IMO, for example you can make websites with good old php 4, you can use appache 1 to host it ... hell you can host it on Windows ME machine if you like. It is working solution and if you know your tools i suppose you will be even faster then most :) But that is not the point.
May be i'm little off to get the point because i have not work with drupal from version 5 and i'm kinda addicted to OOP solutions but still think that this Initiative is the right track for the project.
About the easy for use and making sites ... Symfony is Framework its full OOP its complex on inside, but you can still make blog or even something bigger in day-two or week ( depend on scale of project).
Using old stuff and saying its easy just because you use it for the last 5-10 years its not right. And sadly i see many such opinions around.
I was very happy to see that Drupal 7 ships with brand new Database layer, the idea for the fields is very cool too. Drupal 7 is usable by itself (as must be as far as i understand CMS) for example drupal 6 its not. For every project on drupal6 i sow and use at least 4 modules to make a simple working solution. Contrib modules are not bad, just CMS should provide basic functionality out of the box other way what is the point to use it over Framework ( excuse me if i'm understanding things wrong)
On top of that Views and Panels are pain in the ass to learn even if you know Drupal, because they hack Drupal to do there job, just because core Architecture don't give them much choice.
I will be happy if someone give better working solution, but no matter what the time to rewrite always come some day.
I don't mean to be offensive, no matter how my statement sounds.
Have a nice day
PS: I'll be really glad to help this to work i should dig more in Core i suppose :D
With the developers I've
That is what I was getting at. I never meant to suggest that anyone was opposed to any change, just that they are opposed to any change of this scale and magnitude, which is what my general impression has been.
Get it on
So maybe the solution is to get the hackish implementation Crell did over the christmas weekend to something more in the likes how it is intended and mess around with it.
See how modules behave, run benchmarks, checking it out from head to toe.
Think Crell and some people willing to dive headlong into this could do that. At the moment the discussion is very hypothetical.
To me Webchicks comment cleared up a lot: At the moment the implementation for the bootstrap could be done and not too many things would have to be changed. This would not be taking a lot of benefit of the Http Kernel, but Drupal would still run pretty much the same concerning API and Hooks and all.
The people who see the benefit of the initiative know they are in the situation they have to prove the benefit, so I guess there is no lack of motivation to get this thing more real.
Life is a journey, not a destination
Indeed, Green light
Webchick++
Crell++
Everyone taking part in this discussion ++
We are all taking about things that don't truly exist yet. Let's continue down this path to see what comes of it. Only then will we know what will be necessary to realize the vision of this initiative.
Software Engineer @ The Nerdery
Maybe Yellow
Green light is great but there where a lot of concerns raised and I'm pretty sure heyrocker is trying to work toward a solution for those by identifying where those concerns need to be addressed. If you're excited about moving it forward(like I am), I think the best approach is more to proceed with caution with an eye on this discussion.
edit: spelling error
Demo module
OK, as requested by a few people above I have put together a brain dump sample module of what a future module might look like. As the disclaimer on the module page says, take everything with several grains of salt, no warranty is expressed or implied, caveat emptor. :-)
Questions, post 'em in reply and I'll try to answer. As I said, it's still VERY rough.
Thank you
Thanks for all the comments in these files Crell. I'm reading over the code and trying to get many more eyes on it.
A few questions that may expose my ignorance:
In ServiceRegistration.php, is registerServices just a way to create APIs for your module or does it truly a way to expose web services that can be used to interact with the business logic of a module?
I see the use of "BlockController", should/will there be "PageController"s, "NodeController"s, and "EntityController"s?
During the course of module development, will we be able to build our own custom routings, controllers, and view handlers?
I don't know where in the code sample provided a place where the module is invoking the theme layer. Has that been discussed yet? Will we be able to use Drupal 7's theme layer unchanged or will we have to modify how information is passed to the theme layer as a result of wscci?
Finaly a note to Crell and the WSCCI team:
Thank you for efforts with this initiative. Keep up the good work and keep moving forward. It sometimes takes a herculean effort to make progress. With the effort that is going on here I think we're making progress on all fronts: Understanding what needs to be built, getting community acceptance of the changes, getting community acceptance of the vision.
Software Engineer @ The Nerdery
ServiceRegistration "Service
"Service" here refers to a code service offered by the Dependency Injection Container. It has nothing to do with web services at all. I agree the name is confusing. Symfony calls these things "services", as does some of the formal literature, but I'm not wedded to it if we want to call it something different.
It's not a controller in the sense that the entity system currently uses the term, which is more akin to a Factory in standard lingo.
How routing works exactly is still up in the air. The initial implementation will probably be similar to the menu_router table, but with more information in it. We still have to figure out how to performantly allow the routing to vary based on things like node type. I don't have an answer there yet. Ideally, though, that information moves out of code and into configuration via CMI. That's TBD.
At least as far as WSCCI goes, we're not proposing any large changes to theme system beyond there never being a render array for a region larger than a block. So hook_page_alter goes away. That probably means less use of render arrays overall, which I'm fine with. And with the Services DIC, theme(...) probably becomes $this->services'theme'; Beyond that, the theme system definitely needs an overhaul but we're not directly working on it. More just forcing it to be broken down further.
Might make sense to not use
Might make sense to not use the Controller term at all here, but go with different variations of "Block".
Would probably make it less confusing for Drupal people.
Calling a spade as spade is
Calling a spade as spade is useful though, if its a spade, and these do look like Controllers. Makes it easier for developers new to Drupal, and the rest of us can get used to naming things appropriately.
Doing too much
I went through all the code yesterday and the demo module today. While it looked really awesome, I sort of felt similar to chx: it seemed very foreign. I was having a bit of trouble pinpointing exactly why, but after discussing it with Ryan Cross today, I realized why:
The WSCCI project has sort of evolved into a a bunch of parts:
- overhauling bootstrap (with HttpKernel etc)
- context
- web services
- dependency injection
- super blocks
- plugins
Many of these are closely related, however I think wscci-symfony-dic tries to do too many at once (basically it does everything there except plugins). I think this is what causes some of the backlash and ultimately makes it very hard to review.
I think it would be very useful to try and split some of this stuff out and incrementally get it into core, one piece at a time. It's tempting to rewrite all of Drupal in one pass, but that's never how we've done things. We need to be able to break into small pieces, even though it's difficult to split it up and we may need temporary placeholders in order to facilitate better integration.
Dries advocated for starting with super blocks and working backwards, but I'm not sure if that's entirely feasible, as super blocks require context which requires overhauling bootstrap. So, how about starting with overhauling bootstrap and using HttpKernel? If we could get a 200 line patch in to do just that, it would be much easier to build consensus. After doing that, it's not a huge step to add a $request->context = new RequestContext(), but that can be done in a follow up patch. After that, we can introduce dependency injection and plugins, both separately. Finally, building on all those components, we can build superblocks and superlayouts, etc.
I think this is, or could be, essentially similar to the phased approach we tried for initially, however with completely different phases due to the research that was done since then. We now have a better picture of what Drupal core might look like, what components will use, and how they'll integrate, so we can make a more accurate battle plan.
I think this comment from a while back by chx makes a good point: http://www.garfieldtech.com/drupal-symfony2#comment-2029. We wanted to start out by unifying context. Not a too difficult problem to solve. Now somehow we want to rewrite Drupal. While it's not necessarily bad to rewrite Drupal, we've strayed quite far from the original idea. To directly solve eaton's problem, we could have a global Pimple DIC that's returned from a context() function and that's that. However, clearly, it's become much more. I think the reason it's become much more is because some people, myself most certainly included, have jumped on it, seeing the huge possibilities this allows us, if only we change x y z and suddenly all of core is gone and rewritten.
It's certainly feasible and possible to rewrite core. We just have to do it one step at a time. We can't start by doing all of the above-listed steps except for plugins. We need to take it one step at a time, at each step of the way solving concrete problems with understandable solutions.
How does that sound? Bad idea?
The crux of the problem
Dmitri, I think your post goes right to the heart of what has had me and a few others tearing our hair out for the past 4 months. Despite our best efforts, people are still confused about WSCCI's scope and roadmap, and I am frustrated beyond belief that anyone still is.
Way way back at DrupalCon San Francisco (and in informal discussion even before then), the proposal was for SuperBlocks. That predated web services even being mentioned. SuperBlocks required unifying request/context information as a prerequisite, and various other things. (Replacing hooks with objects was discussed early on, although at present I don't think is on the table). A multi-step approach was always, from the beginning, the intent so that we did not "rewrite all of core at once". chx's comment on that blog post is incorrect. Unifying context is the only part he bought into, but most other people I talk to (like 90% or better) fully support the entire process and WSCCI has always had more on its roadmap than just unifying the request object.
When we tried to implement just part 1 though, unified context, we got a ton of pushback. That pushback came down to two points: 1) There's too much here and it's over-engineered (because we built in stuff we expected to be useful later on) and 2) Without seeing the end system in place I don't know what the use case is.
That is, "I am looking at a tree, and don't understand the forest" (even though the forest has been explained sans-code-that-doesnt-exist yet many many many times).
So, I put together this proof-of-concept. The wscci-symfony-dic branch was never, ever intended to be something we merge straight to core. It was a skunkworks proof of concept to provide an answer to the "so what does the end system look like?", leveraging Symfony further than we had been because, as I discovered, they were already most of the way to where we want to be.
The pushback we're getting now is "OMG too big", "can't we just do it piecemeal?", etc. That is, "The forest scares me, please just show me a tree." Your comment is a perfect example of that.
Hence the hair pulling, because we do not have an infinite amount of time to Goldielocks and guess what size of work will be enough to get people to calm and and say "OK, just right". I simply do not have that kind of time or energy, and neither does anyone else. Nor, really, do we have the time or ability to crowdsource the development of that size. It just doesn't work. Roadmaps and implementation plans don't work that way.
I want to develop a new phased implementation plan, but quite simply, unless there's a clear understanding that the end goal is acceptable and that working toward that goal is not going to run into the same forest vs. trees problem every single issue, it's not worth the blood, sweat, and tears to do so. We cannot bikeshed the forest when discussing each individual tree, or the forest will never get planted.
You eat an elephant one bite at a time, but you'll never finish if you stop to ask "should we really be eating an elephant rather than a giraffe?" every single bite.
I think I've run out of metaphors now... :-)
In short, yes, we want to take incremental steps toward this end goal, and the end result will no doubt not be quite the same as what's presented here. But there's no point in taking those incremental steps if that end goal won't be accepted.
And it certainly seemed until November or so that the end goal was. Now, after seeing the first tree, there's lots of pushback against the forest. And I simply don't know what to do with that.
Yes
I see completely what you mean. I don't know what to say, because that's a problem that I have no idea how to solve. Do you have any ideas about where to go from here? Maybe bring that point up with Dries/webchick?
(nice metaphors, BTW!)
Don't give up!
When a message is having difficulty being disseminated, often a good thing to do is to break it up into smaller pieces. Maybe break it up into a "D8 WSCCI" and a "D9+ WSCCI". Cover the forest in D9+ and afterwards start picking trees for D8. And/or, if there is one thing people are consistently not getting, write it down, then copy-paste into each relevant post you make, as a disclaimer, or even put it in your signature (or a link to it). Just a thought :)
This is great work you're doing -- I don't think anyone is truly against you or the WSCCI (or at least not many), just a matter of breaking down the message.
sethviebrock.com
There's a bunch of people
There's a bunch of people helping out with the WSCII rewrite, which is great. They're pretty much in support. There's a bunch of people watching the rewrite; they might occasionally help with suggestions and a patch here or there, but they're also concerned from time to time, holding back, and commenting with their perspective and worries.
Then there's a bunch of people who see or hear about the work going on, are broadly in favour, but don't necessarily have time to pitch in. So they're quiet, and you don't necessarily notice them. It might feel like you're getting lots of pushback, but don't get discouraged - you're always going to hear more from the people in opposition than the people in favour.
(Which isn't too say I would discount the critics: disagreement is healthy and their suggestions or comments might make for a better end result.)
Formula One makes iterative improvements throughout the season: they add new wings, tweaks to the bodywork, new components. But every year they also start afresh. Incrememental changes can makes things simple, but it's usually difficult to make revolutionary changes in incremental steps.
Personally, I'm fully in favour. I just wish I was fully up to speed with the changes and had more free time to help out.
--
Marcus Deglos
Founder / Technical Architect @ Techito.
great metaphor
I have heard this before, but just wanted to point out how much i like this metaphor:
--Ryan
Ryan Cross
Drupal Development Services
ProjectPier project management and collaboration software
class diagrams
Hi
I just joined this group after hearing Dmitri speaking about it at DDU on the weekend. I don't have a complete grasp of the scope as yet. After reading the past couple of messages, however, my suggestion is (if you aren't already doing this) that you really look at a proper software design process. It seems to me like this is not one of those projects that you can vaguely define it, partition out, and hope it all comes together in the end. You need a process, and countless computer science and software engineering books have been written on the topic (the one I'm reading at the moment is The Design of Design by Frederick Brooks). If you're not sure where to start, try the spiral model and refine your design iteratively.
There's also an effort to make Drupal more OO, which I'm totally in favour of. If D8 is to be consistent then WSCCI should probably be structured around an OO architecture. Hence, you need to start with UML class diagrams showing classes for things like Block and SuperBlock and Context and whatever other classes will be in the data mode, and the relationships between them. Class diagrams are a great way to get a picture of your app and how everything is related. It gives you a framework, not only for your classes, but also the underlying data model.
Sorry if this has all been said before. As I said, I just joined this group and haven't read everything yet. Just throwing in my $0.02.
Cheers,
Mossy
Drupal + PHP community
Hi,
I've been reading this thread and thinking my own position in the whole mess of things. It might be a bit rambling but nonetheless I hope someone finds the grains inside it.
So let's start first from my viewpoint. I work in a company where our only focus for now is building kick-ass websites. We don't design them, we don't do any conceptual work even if we could do some. We create the technology behind every website, and we rule in that area. We do have people capable of challenging ideas that clients have if they seem funny/illogical to us but that's something the client get for free by working with us.
Since we have a technical background, we want to rule in that side of things, we want to be the best. We operate in Finland where the largest website projects are not that large in the grand scheme of things but they offer us nice challenges nonetheless. To mess up our position more we have two teams operating under us. We have a purists and then we have drupal people.
Purists as I now call them write PHP, they hate drupal since it's too dependent on the database. Configuration lives in database (seriously?) and deploying new versions of code and config is utterly insane when you have configuration and content mixed in one database. So they operate using Zend Framework, Symphony, Doctrine, they use MongoDB if applicable and they live in the bleeding edge. Do you know how much shit we (drupal people) take from them when we start describing our functional based software and deploying procedures. A lot, but we manage since we like how drupal functions and helps us when starting to build a website.
What I hate about drupal? Everything that current initiatives are trying to fix. I absolutely freaking love configuration initiative (deployment using only versioncontrol yay), but more so I love what WSCCI has become of.
We have these things in drupal called modules, and everything evolves around them. We have this utterly legacy code that lives in includes/ folder. We don't use OOP, we don't use namespaces, we don't use any new technology that we could if we wanted. Because of backwards compatibility and legacy purposes. New drupal versions have been built around the idea that API can break and we can include a ton of shit as long as it's good shit. I'm having mixed feelings about people complaining that they need to relearn drupal.
In my opinion drupal's biggest fails if not taking this technical legacy we have, is it's architecture is so different than other "actual" frameworks. If we can move more towards what symphony2 framework is like. Utilize the naming conventions they use, start using namespaces, refactor the hell of everything and ditch hooks in favor of classes and plugins. Why not? If we lose 50% of drupal people in the process, we gain 100% of the rest PHP community.
Do you know how hard it is to start teaching new PHP people drupal's architecture, it's a huge task and people who have been using drupal for two (2) years are still finding new things how to do stuff. It's utterly insane in my opinnion. Why do we want to live so far away from rest of the PHP community when we could start living next to them.
So all this said and done. I'm all in favor of giving Crell the thumbs up, build everything on top of S2, refactor the hell of drupal's core. Use existing S2 / Zend components. And let's make Drupal 8 something that PHP purists can enjoy using and building on top of (and continue this in D9). The danger if we don't do this is that Symphony-CFM starts to gain traction and by the time we get to Drupal 9 all is lost because PHP and the whole Internet has evolved into something that drupal doesn't can't handle anymore. Sure there will be some follow up of people but it just doesn't matter if the technology is lost on something superior.
And those who start crying about the amount of module rewrites that need to be made, so what? Do you remember the time when D6 was released and you needed to wait about 6-12 months before getting a working version of CCK, Views and Panels. You couldn't build anything fancy without those so D6 adoption was really slow. Yes it will be again really slow, but at least we as a community are moving towards what every other PHP developer is accustomed to, not moving away from it.
Cheers, Santeri
I'm still distracted by how
I'm still distracted by how awesome the jedi ninja assassins were, so I misssed most of the conversation.
But Crell is a jedi ninja assassin, and if he says the Symphony way is better, I believe him. The best things about Drupal 7 are the things based on ideas we stole from elsewhere (DB:TNG). The worst things about Drupal 7 are based on our own sordid history (Render API). I say we steal more stuff.
Also fogenrator/Santeri is right on. I've built a business on turning PHP people into Drupal people and (a) it only works on relatively inexperienced PHP people (hmm...) and (b) I'd be happy to let that part of my business die so we can all move on to better things.
Drupal.org user profile
Drupal Micro-blogging: http://twitter.com/matt2000
Hey!
I resemble that.
Also, agreed on the front of training. Without the galoshes of experience, getting your feet wet in Drupal on the front lines can mean being waist deep in the piranha swamp. New devs make crazy mistakes and end up in the theme layer because it's the closest to changing the output they've got that almost makes sense, and yet it just creates booby traps for their coworkers and even themselves only months later once they start to get it.
Same sad song we all know. I'm 100% for re-learning a few things so that the platform can become better, and the learning curve can be pulled back a bit for new devs.
Two cents
I am absolutely newbie in this scenario but I feel two things in all honesty
1) Maintain drupal 5 and 6 with nominal support - thousands, if not millions, of sites will continue to live in those for eternity as long as hosting lets them survive - they will be having mild or moderate number of visitors or be just archival as the creators/hobby webmasters/etc will become old and not be uptodate geeks.
2) Start a NEW drupal NOW! Incorporate whatever is new, whatever new code, whatever new user expectancies of the current users that are accustomed to FB and G+, and whatever really n e w new things that drupal can show for the rest of the world to follow.
Incremental release of 7-->8-->9 may be just out-of-date and waste of precious human resources.
At the same time not killing drupal 5 and/or 6 will be eternal blessing to those who help the "geriatric" not to be sent to old-age home.
.
Since you're new, you may not realize it, but this is just not gonna happen. Drupal 5 is already unsupported. There are many, many discussions about the whys and wherefores on this already if you do a bit of searching. And this is really off topic for this discussion.
Michelle
We are
Between the Configuration Management Initiative and WSCCI, Drupal 8 is the "NEW Drupal". Or at least, that's the goal.
That is exactly what is so scary to some people, and why it's been such an uphill battle to even get acceptance on the roadmap and battle plan.
Legacy support for older versions that are already retired is entirely irrelevant to that question, and as Michelle said is not even remotely on the table.
As an outsider I think the
As an outsider I think the key questions are:
1) is there a problem that needs fixed?
No: then stop here
Yes: then
2) do you want to fix it?
No: then stop here
Yes: then empower those who wish to spend the time to fix the issue, rather than keep them busy with endless discussions like this
BTW, saying No to 1) or 2) is entirely legit too. If only a minority think there is a problem or that its worth fixing, then say so, then these people who do see an issue that they want to fix can look else where.
That being said again as an outsider I wonder if there is any decision making process? Reading this thread it feels a bit like the winner is whoever gets tired replying first, while with the initiatives owner concept it initially seemed to me like these people are empowered to make decisions on their past track record.
Anyway, again only the view point of an outsider that obviously is clueless of the inner workings of how Drupal is developed.
LTS project
While Drupal 5 is no longer officially supported by the Drupal project or Drupal security team, and there's no serious plans to change that that I'm aware of, realize that as an open-source community, people who care about this can grassroots organize to solve a shared need without any "official" authorization. Here's one project attempting to do so, http://drupal.org/project/lts, which also links to http://groups.drupal.org/drupal-lts for organizing discussion. However, those don't appear very active to me, so either the need isn't that widely felt, or people don't know about them. Anyway, out of scope for this thread, so for anyone interested in continuing a conversation on D5 support, please do so there.
2 cents + 2 cents = ...
Lot of people throwing their two cents, so this discussion is getting rich :)
I like the fact that people are supporting the initiative and the evolution of the web (from both a dev side and user side). Drupal too will get richer.
gaining steam
This clearly evolves into polarizing.
Pole 1: leaving the "not invented here" Syndrom behind and embracing the mainstream of PHP development - if that exists. Hopefully - but not for granted - making it way easier for Devs to learn to code for our platform. Also leaving behind of beloved handcrafted solutions that work very well but somehow reinvented the wheel.
Pole 2: feasability and doability. How to not alienate people who have already learnt drupal, written modules, built businesses upon it that are not so small?
I guess there are examples for similarly bold changes in software projects that failed with great hello or failed with very silent agony.
And there are some that were incredibly successful.
What makes Drupal Drupal? Mostly I guess stuff that is one level more up in Catch's pyramid he depicted once. Entities, the way we leverage them, Fields, Views and how every "modern" Drupal module utilizes the granular API-like modules, ultimately being abele to build even "Apps" and Distributions that definitely differ from other platforms. Because the granularity and control is still in your hands when you need them. And most of all the community that is for the most part working together in a very constructive way.
So this level is not threatened by even a total rewrite of Drupal I guess, should be able to keep it completely no matter how wild this could get.
But.
Even If everyone would agree refactoring as much as possible would be the way to go, we need to get a release out in say about two years. Drupal 7 took a year longer than expected and this was a real threat to the professionalizing the release process and being more reliable and handlable by business partners. I do not know how many grey hairs Dries (and others) grew in 2010 over that. Also a not small burn-out syndrom in the very hardest working core devs is still quite fresh in memory I guess. In the end it is not so many people doing all the heavy lifting and endless debugging. Time and sheer physical endurance are not endless.
So - is it possible to do a more long-term plan? If there was an agreement on the basic approach (hard enough to find that), could D8 build the base and put into reality the first phase, and leave just enough of the existing architecture in. Just enough so a release can be done in two years, but it still does not compromise a somewhat stringent approach to build on it further?
A bit of a problem I see in that is that Drupal is no top-down product. That's not how it works. Do we have the organizational and project-management structure in place to pull something like this off? I somehow doubt it. So maybe find an even more granular approach, which is what I guess Larry and the more pragmatic proponents of the entire Symfony2 thing are after. Some years of experience in the project tell a bit of a different story of what is doable and can be debugged and put into place in a given amount of time.
But as this is also rightfully dreaming up an architecture that we would like to see, I'd say that for the moment the sky is the limit, at least in discussion. Really coding it and getting patches into core will come down to reality more quickly than one would wish :)
Life is a journey, not a destination
First thoughts on Symfony-DIC (WSCCI-TNG)
Hello all,
I just wanted to give you an update on where I'm at with WSCCI. The past weeks, I've spent a good chunk of my time reading up on the various discussion threads, reading code, talking to Larry and other people in the community.
Given the size and impact of these decisions, I've more brainstorming and reading to do. To do so, I've invited several people to a 3 day WSCCI code sprint that I'm organizing with Angie's help.
Also, I've freed up time in my schedule to do a deeper dive on the code and to comment on the proposed architecture. I allocated 3 hours of time tomorrow, and have much more time set aside the coming weeks. Expect me to start commenting in the WSCCI issues.
This thread is a bit more meta, so let me share my current meta thoughts here. All of these thoughts are subject to change or evolution, but it is probably good to do an interim braindump.
To me, the big question is not "how" we best to do this, but "why" we best make this change. After much thought, that translates to: "How do we win the hearts and minds of even more developers?". Getting more people into the Drupal project is the biggest challenge that we're faced with.
We may have arrived at a point where we alienate both the "self-taught hacks" (no disrespect implied) as well as the "formally trained developers". Drupal is too complicated for the former, and too unconventional for the latter.
Symfony-DIC (WSCCI-TNG) gives us the opportunity to go after the formally trained developers, and eventually, to get more people involved into Drupal. Hence, I'm warming up to Symfony-DIC (WSCCI-TNG).
And it is not "really" about the low-end versus the high-end. There are a lot of small sites developed by skilled developers. In fact, most of my webby friends work on small sites/projects and they are both skilled and experienced developers. The ones that are junior developers aspire to become skilled developers working on big projects. They absolutely want to learn how to use the tools that skilled developers use. In today's world, that means using a modern object-oriented framework based on the principles of MVC.
My Dream
I dream of a future where I can have a professional PHP developer dive into a Drupal project an understand 80% of what they see. Adopting standards that were forged outside of the Drupal context may go a long ways to making that a reality.
Software Engineer @ The Nerdery
I was with you until the last
I was with you until the last three letters :) More seriously, I'd pictured Drupal ultimately formalising its AOP approach and I'm a bit peturbed by the fact that not one of the many MVC frameworks in different languages has produced a popular generic modular CMS. I've seen a lot of attempts, and as a former Zend developer who brought Drupal into a large Zend/Symfony company (where it massively reduced time to market), I always felt it was an architectural issue stemming from Drupal using PAC/AOP rather than MVC. I could be wrong and this turns out to be an underlying implementation detail that is solvable by the right community meeting the right API architecture, but seems worth understanding why MVC hasn't succeeded in this area (or where it has come closest), and why PAC/AOP seems to shine here.
I'd agree with you - moving
I'd agree with you - moving Drupal away from it's architecture of relatively "dumb" display layer would a bad step.
Larry's comparison is probably still the best reference: http://www.garfieldtech.com/blog/mvc-vs-pac
I can't imagine that Larry is suggesting a move to MVC?
Ha!
I was wondering if anyone else was going to catch that. ;-) Yeah, what we're proposing here is neither real MVC nor web-MVC/Model2/whatever the heck Sun and Rails misnamed that damned thing. It has something called a "Controller", but that doesn't make it MVC.
If anything, it would be moving Drupal more toward "proper" PAC than our current quasi-PAC. Traditional PAC does have layering to it, or at least multiple PAC triads, which would in this case roughly correspond to the nested Block controllers we're talking about.
What we would be moving to, however, is more industry-proven methodologies, ways of thinking and approaching problems, and so forth. That's a non-small shift for Drupal to make. But it is, I firmly believe, 100% necessary that we change our approach and accept that, you know, we're not the first in this space, we're not alone in this space, and we've got a lot we can learn in this space. That doesn't mean we stop being Drupal. It means we stop being an island.
Dries is absolutely correct about the impact this has on new developers. Right now, I don't think I could name one single person who likes the Drupal code codebase and architecture who was not weened on Drupal. And even many of them are not fond of its current design. The ramp-up process for new developers is extremely steep, precisely because we do things in such unorthodox ways. Sometimes there are good reasons for that, sometimes there are simply historical reasons for that, but in either case it makes Drupal harder to learn for anyone who isn't learning programming by copying and pasting Drupal code. If, however, we bite the bullet, swallow our pride, and accept that sometimes there's a reason the road less traveled is less traveled, we could open up to a wide range of potential contributors.
OO, PSR-0, dependency injection, lots of chained method calls, and all that jazz may seem unusual to the 800 or so of us that are embedded in Drupal. But to the thousands of people who are used to PEAR, Zend, Symfony, Lithium, or anything Java, it's more like a warm bath. We want those people to be able to make the switch to Drupal when their company decides they want a Drupal site, arguably even more than we want the proverbial "kid in his parents basement". (Most of them are now starting off with those techniques, too.)
Drupal is, in a sense, the last great PHP 4 framework. We need to charge forward and become one of the great PHP 5.3 frameworks (and soon, PHP 5.4). I think we can do it. Really, I do. But only if we step back, take a deep breath, and give ourselves the freedom to say "you know, what? This will be hard and confusing. Life will be different after this. But that's OK, because it will be a better life." And then charge ahead at full speed, because otherwise we'll never get it done.
I don't think the question is can we... it's will we.
As a "self taught hack", my
As a "self taught hack", my life became much easier when I started learning common design paradigms. Sometimes this has meant having to bend Drupal to those patterns, not the other way round, but my life is easier for it. For example, OOP seemed daunting at first, but is actually a lot easier than the same thing done procedurally. If we lose the "hacks" because they are unable to adapt, then probably they are not the developers we wanted anyway. I don't think this will happen, however, as long as we ensure we document and educate well. Most people can adapt when they can see how to get to their desired outcome.
I'm also self taught. I was
I'm also self taught. I was a business major in college. I learned Drupal programming first before general programming concepts. As I progressed in my education learning about design patterns, I can appreciate the what the WSCCI movement is doing.
Neil Hastings
http://twitter.com/indytechcook
Let's talk about the code
It would be great to see this conversation focus as much as possible on WSCCI and the actual code under consideration. I don't think it's helpful to make generalizations about different kinds of software developers and what they're capable of or inclined to learn, and this thread seems to have too much of that (which is ironic, because this thread is also full of people who break every single one of those stereotypes; you can even see that in some of the anecdotes above).
If we want to make a big push to improve Drupal's developer experience, that's not the topic of this thread, so let's do that separately and do it the right way - which means (following the model used for user experience changes in the past) we need to do some testing, get some actual evidence about the problems new Drupal developers face, and make changes specifically designed to address those problems. We shouldn't be doing it based on random anecdotes.
The main goal of WSCCI, however, isn't to improve developer experience; to the extent that it will naturally have an effect on developer experience it needs to be evaluated on that, but only to that extent. Its actual goals lie elsewhere. Also, when I look at Crell's sample module at http://drupal.org/sandbox/Crell/1398226 (which only a few people have commented on in detail after he took the time to post it) I find it hard to imagine how this would either scare away tons of current/inexperienced developers or bring a massive influx of new developers in. Even if this change is achieved in its entirety (which it's not likely to be, because few things in Drupal ever wind up looking like people think they might look many months in advance), the API changes simply aren't that transformational. It still has hooks and the form API and the theme system just like Drupal modules have today. There are a number of small changes, and a few decent-sized ones, but none of them are huge.
Let's look at some of the decent-sized changes that module developers would face and that are specifically associated with WSCCI:
By itself this is not really that different from Drupal 7. Possible complexity comes if your module defines blocks with highly dynamic and interrelated subjects and content (like aggregator module); in that case having the return value be an array rather than two separate methods is probably more convenient, but overall this is an implementation detail.
Where this does become interesting in terms of developer experience (and here I'm paraphrasing a concern I heard webchick raise) is that currently in Drupal pretty much everything is a hook; if you want to respond to something Drupal does or provide Drupal some information, hooks are almost always the way to do that. If we have hooks and controllers/classes then people have to be able to figure out which pattern they're going to follow in each case. That could get complicated.
Fortunately, though, the effect of this on developer experience should be pretty easy to test in advance. We don't need working code, just a list of what is intended to become what. Take some developers and give them a list of ten things that are all hooks in Drupal 7, and see if they can correctly figure out which would be hooks and which would be classes/plugins/controllers in Drupal 8 (and why). If they can figure it out easily that means the concept is understandable; if not, the distinction needs more work.
$request->getPathInfo()may be a substitute forcurrent_path()in Drupal 7.This concept is nothing new to Drupal developers; anyone who has ever used the form API is already used to having $form_state passed around to every function, and that's the same basic idea (it contains contextual information about the current state of the form). Other less common examples already in Drupal include JavaScript behaviors, which always have a "context" parameter passed to them (as pwolanin alluded to above), and the Drupal installer API, which passes an $install_state parameter around everywhere as well. The only thing really new is that it's being unified and extended to more places.
In fact, one of the things I was surprised to see in the sample module is that $form_state is still there; I would have guessed this would be part of $context or $request or whatever, and that would be passed in to the hook instead of $form_state. If something like $form_state is not possible or easy to convert to the new system (and it might not be) then that's a point worth discussing.
t('some text')you would call something like$services['translation']->t('some text'). Besides being more typing, this isn't a huge change; however, if your module has an API you might (in some cases) also have to define services also, and then other modules which use your API would do$services['someconcept']->someFunction().I think the impact and benefits/risks of this proposed change need more discussion. It also does not seem as closely related to WSCCI as the others.
The biggest changes being proposed are of course not to modules but rather to Drupal core internals, but on some level those tend to get rewritten every major release anyway (at least parts of them). One thing that we might need more of here is a discussion of whether this approach is really the simplest and smallest change to Drupal core that would accomplish the overall end goal. We shouldn't be afraid to make changes (even big changes), but we shouldn't do big changes unless there's a really good reason. Drupal is an very successful software project, so it's OK to be a little conservative about it.
Thank you!
David, thank you for actually reading my code. :-) Some feedback:
Well, remember that a block is simply a special case of a controller, which is a more generic case of "thingie that returns a response object". It actually has an execute() method in the interface. That all happens further up the object hierarchy. Have a look at the symfony-dic branch in the main wscci repository and you'll see what I mean. All method names are subject to change, and EclipseGc has been doing his own POC that works a bit differently. We need to reconcile those at some point.
The separate subject() and run/body methods don't mean the code can't relate. It simply acknowledges that at an API level, those are things you should request separately. Within the object you can dynamically derive those however you want, including kicking both methods off to an internal method that generates both together and returns an array that each one uses just its part out of. That's probably unlikely to be a good approach, but it's always there. Proper interface-driven development gives you a TON more flexibility than we have now.
That's true, but I think that's actually a problem to solve. Hooks are our sole hammer, so we use it even when we have screws. A lot of things in Drupal that we use module_invoke() on, or a manual callback, really should not be hooks at all. Hooks are, properly, an event system. That's all. We just happen to contort our event system to also be a plugin system (kinda), a data definition system (sort of), a non-class-keyword-using form of OO (far too often), etc. Controllers are more analogous to page callback functions, but 10x more flexible and powerful. The more generic replacement for most not-really-hooks-but-we-pretend-the-are-to-confuse-people is plugins. EclipseGc recently summarized those as "all those places that you have an info hook now? Yeah, those are almost all plugins."
Hooks are to respond to an event. Plugins are to provide reusable functionality.
The classes used to hook up the service object and extended request information look kinda like hooks, I agree. For the extended request information that's because Symfony2 works that way. That's its own event system at work, so it's not surprising that it bears some resemblance to our event system. For service objects, eh, I needed something that would work without full bootstrap for the time being and it's just a proof of concept. :-) I don't know yet what the final solution should be there.
See http://drupal.org/node/1417320 :-)
That said, the idea is not to have one single "state" object that everything uses in all places. That would not be much of an improvement. The request object is just that; information about the HTTP request that caused PHP to begin executing. The service object is just that; a container for various "things that do things for you" in Drupal. The form state is just that, the state of a form. Those should not be folded into a single object, or we end up with "one giant array of doom" again that no one can understand.
Correct. The point here is that we need the extra layer of abstraction that an OO service offers us, rather than just throwing functions around, to:
I actually agree entirely here. :-) In this case, I started off 2 years ago with the question "what would it take to add multiple layouts to Drupal, like Panels offers? Just that, nothing else." Unfortunately after studying the problem for a while I came to the conclusion that the only way to do that cleanly, not hacking in something somewhere with yet another undocumented alter, would be to reach down Drupal's throat, grab hold, and pull until the entire page building process was reversed. Along the way, web services and other non-HTML responses become really easy to add as well. There is, unfortunately, no way I've found in 2 years to do those well without this level of change, because Drupal is so hard-architected for "initialize, load everything, generate a huge blob of HTML, print, exit" that you really can't get it to do anything else in a non-hacky way.
The Panels team has been trying to deal with this problem for years and years now, and the result is still effectively a completely new router system sitting on top of core's plus a UI that makes no sense because it has to straddle two vastly different routers. There's no reason for us to have a CMS built on top of a CMS just to get decent page layouts. The fact that the entire Panels/ctools team is on board with these sorts of changes to core is, I think, very telling.
Update
JFTR, in order of appearance: