There was a meeting last week between Crell, the Palantir team, webchick, and chrisstrahl to discuss the prospect of funding the WSCCI initiative. This meeting resulted in a funding proposal to resource this initiative through the Large Scale Drupal program and the Drupal community. This page and the attached documents are the results of the discussion. If you're interested in getting involved, please, contact me and I'll fill you in with further details of how to get involved.
As the web matures, content management systems need to output data to many more devices and integrate with more and more systems and services. Today, Drupal is optimized for outputting HTML and the core system ships with an antiquated XML-RPC backend. Designing Drupal for the future of the web includes revitalizing and improving these systems.
The goal of Drupal 8's Web Service Initiative (also known as “WSCCI”) is to make Drupal equally good at outputting data as XML, JSON and other non-HTML formats through a web services (RESTful) interface. We also want to provide more robust support for modern web caching technologies such as Varnish and Edge-Side Includes (ESI). To support these capabilities, several parts of Drupal’s older PHP 4.x-based procedural framework will be swapped out with components from Symfony -- a modern, object-oriented, PHP 5.x web framework.
Drupal 7 and previous versions assume that each request will return a fully themed HTML page, which is fine for most desktop browsers. However, doing something other than that—for example, returning XML data for a mobile application, or JSON options for an autocomplete field, or even just a page fragment—requires workarounds, such as stopping the page request before the theme system can take effect. Additionally, the theme system (and other systems such as the Form API) are nevertheless loaded on each page, regardless of if they are used.
The diagram below outlines the variety of ways of a request can be translated into output in Drupal today. Notice that although the request only comes in once, there are no less than three different systems to determine how Drupal will handle responding to the request.
Many sites abstract this complexity by building on a “middleware” foundation such as Panels, Context, or Services modules. These middleware layers can provide additional capabilities that Drupal core lacks, such as mapping additional contextual information (location, content being viewed, current user, HTTP information) on a request.
Unfortunately, these middleware layers introduce additional risk because they are supported by a much smaller maintainer team, tested by a smaller subset of users, only have partial support in other contributed modules, and are largely incompatible with one another. They also create an additional maintenance burden on companies that rely on them, which adds up to significant costs over the lifecycle of the implementation. Using any one of these middleware systems forces implementation of an architecture limiting the ability to utilize all functions available in Drupal core.
In Drupal 8, the Web Services initiative hopes to leverage the Symfony framework in order to radically streamline the request/response cycle, and remove base assumptions in Drupal’s request handling that make little sense in a world of content being deployed in multiple formats to multiple devices. The goal is to “future proof” Drupal to accommodate future of web and make it more platform-agnostic while building the capabilities currently provided by “middleware” components directly in Drupal core.
The revised vision for the request/response diagram from Drupal 7 in Drupal 8 is pictured below. Drupal 8 aims to offer much more streamlined requests and offer much better consistency than Drupal 7. Each request, regardless of whether it’s an HTML page destined for a desktop browser or an XML feed destined for an iPhone app, follows the same basic code logic path, whether it’s a request or a sub-request.
Currently, WSCCI’s efforts have been focused around proving the design and architecture of the system through research and prototyping. Several in-person gatherings have been held with key people within both the Drupal and Symfony projects to come up with architecture requirements. What lies ahead is implementation in Drupal core of major components that enable web service features.
There are currently a few major patches in progress to Drupal core, but they would greatly benefit from dedicated, focused time from the initiative lead, Larry Garfield. Larry’s efforts would be focused on getting these basic components completed to enable others in the community to build from a solid foundation. More specifically, this foundation work will remove blockers to work in other areas of core development including the Blocks and Layouts initiative. This also allows for more crowd-sourcing opportunities with the Drupal community to work on the remainder of the initiative’s tasks.
The Web Services Landscape
This project is resourcing only a portion of the overall Web Services Initiative. It will provide foundational architecture that enables future Drupal 8 work, including improved performance via Edge Side Include (ESI) support, a flexible layout and block system, and better support for mobile devices.
The diagram below outlines both the work that is part of this proposal, and also summarizes remaining parts of the Web Services Initiative that are dependent upon this work.
In order to accelerate the velocity of the Web Services Initiative, the funding is being requested for the following key issues:
A) Drupal Kernel Patch (http://drupal.org/node/1463656). This restructures Drupal to use the Symfony request/response delivery flow as diagrammed above. This code will simplify and streamline the current complex system allowing developers a clear, consistent, and simple way to manage requests and responses in Drupal and enable communication between Drupal systems. This is an essential foundation to all future work in this initiative.
-- Development and discussion (3 weeks)
B) Routing System (http://groups.drupal.org/node/220269): This is required plumbing to allow for a Drupal request to return a themed HTML page to a desktop browser and an XML or JSON representation of the same resource to an iPhone app. It would also enable more robust Server-to-Server communication over a variety of REST-style protocols. This would happen automatically based on the incoming request. It provides underlying functionality to turn Drupal into a first-class REST-based web services framework.
Put more technically, this patch will extend Drupal’s current request routing system to include the ability to perform different logic depending on the HTTP method (GET, POST, PUT, etc.) acceptable mime type (text/html, application/json, etc.), and restrictions on the datatypes of parameters (int, string). This is critical, because it allows Drupal to reliably handle web services.
The Kernel Patch must be completed prior to work beginning on this system.
-- Development and discussion (4 weeks)
C) Canonical Format Discovery (http://groups.drupal.org/node/197583): Syndication of content from one Drupal site to another requires a common format for communication. A standard canonical export format opens up a wide range of possibilities for deployment, alternate storage, content syndication, and web services for Drupal-specific mobile apps. This discovery project will define this canonical format and the methods for implementing it within the Drupal project.
-- Discovery, discussion, and planning (3 weeks)
D) Sprint funding
Larry Garfield will be in attendance for Symfony Live in Paris (June 6-9). A sprint the previous weekend to discuss and plan a direction for the canonical export format would be valuable to form a clear direction for such work. Recommended attendees would be Larry, Wolfgang Ziegler (fago, Austria), Dick Olson (dixon_, Qatar), Yves Chedemois (Paris), and Damien Tournoud (Paris). That would likely be a 3-day sprint, and could be anywhere in the June 2-6 range, depending on travel schedules. Funding would include accommodation for Larry, Wolfgang and Dick; flights for Wolfgang and Dick.
Additional background reading, for the curious:
- Web Services and Context Core Initiative group: http://groups.drupal.org/wscci
- WSCCI sandbox (code): http://drupal.org/sandbox/Crell/1260830
- Original Web Services initiative announcement: http://buytaert.net/web-services-in-drupal-8 http://www.garfieldtech.com/blog/web-services-initiative
- Revised Web Services initiative announcement: http://garfieldtech.com/blog/refocusing-wscci http://buytaert.net/the-future-is-a-restful-drupal
- Overall goal of WSCCI explained: http://garfieldtech.com/blog/drupal-post-page-era