Drupal 8 Web Services Funding Proposal

We encourage users to post events happening in the community to the community events group on https://www.drupal.org.
You are viewing a wiki page. You are welcome to join the group and then edit it. Be bold!

Introduction

There was a meeting last week between Crell, the Palantir team, webchick, and chrisstrahl to discuss the prospect of funding the WSCCI initiative. This meeting resulted in a funding proposal to resource this initiative through the Large Scale Drupal program and the Drupal community. This page and the attached documents are the results of the discussion. If you're interested in getting involved, please, contact me and I'll fill you in with further details of how to get involved.

Overview

As the web matures, content management systems need to output data to many more devices and integrate with more and more systems and services. Today, Drupal is optimized for outputting HTML and the core system ships with an antiquated XML-RPC backend. Designing Drupal for the future of the web includes revitalizing and improving these systems.

The goal of Drupal 8's Web Service Initiative (also known as “WSCCI”) is to make Drupal equally good at outputting data as XML, JSON and other non-HTML formats through a web services (RESTful) interface. We also want to provide more robust support for modern web caching technologies such as Varnish and Edge-Side Includes (ESI). To support these capabilities, several parts of Drupal’s older PHP 4.x-based procedural framework will be swapped out with components from Symfony -- a modern, object-oriented, PHP 5.x web framework.

Current Limitations

Drupal 7 and previous versions assume that each request will return a fully themed HTML page, which is fine for most desktop browsers. However, doing something other than that—for example, returning XML data for a mobile application, or JSON options for an autocomplete field, or even just a page fragment—requires workarounds, such as stopping the page request before the theme system can take effect. Additionally, the theme system (and other systems such as the Form API) are nevertheless loaded on each page, regardless of if they are used.

The diagram below outlines the variety of ways of a request can be translated into output in Drupal today. Notice that although the request only comes in once, there are no less than three different systems to determine how Drupal will handle responding to the request.

Many sites abstract this complexity by building on a “middleware” foundation such as Panels, Context, or Services modules. These middleware layers can provide additional capabilities that Drupal core lacks, such as mapping additional contextual information (location, content being viewed, current user, HTTP information) on a request.

Unfortunately, these middleware layers introduce additional risk because they are supported by a much smaller maintainer team, tested by a smaller subset of users, only have partial support in other contributed modules, and are largely incompatible with one another. They also create an additional maintenance burden on companies that rely on them, which adds up to significant costs over the lifecycle of the implementation. Using any one of these middleware systems forces implementation of an architecture limiting the ability to utilize all functions available in Drupal core.

Future Directions

In Drupal 8, the Web Services initiative hopes to leverage the Symfony framework in order to radically streamline the request/response cycle, and remove base assumptions in Drupal’s request handling that make little sense in a world of content being deployed in multiple formats to multiple devices. The goal is to “future proof” Drupal to accommodate future of web and make it more platform-agnostic while building the capabilities currently provided by “middleware” components directly in Drupal core.

The revised vision for the request/response diagram from Drupal 7 in Drupal 8 is pictured below. Drupal 8 aims to offer much more streamlined requests and offer much better consistency than Drupal 7. Each request, regardless of whether it’s an HTML page destined for a desktop browser or an XML feed destined for an iPhone app, follows the same basic code logic path, whether it’s a request or a sub-request.

Current Status

Currently, WSCCI’s efforts have been focused around proving the design and architecture of the system through research and prototyping. Several in-person gatherings have been held with key people within both the Drupal and Symfony projects to come up with architecture requirements. What lies ahead is implementation in Drupal core of major components that enable web service features.

There are currently a few major patches in progress to Drupal core, but they would greatly benefit from dedicated, focused time from the initiative lead, Larry Garfield. Larry’s efforts would be focused on getting these basic components completed to enable others in the community to build from a solid foundation. More specifically, this foundation work will remove blockers to work in other areas of core development including the Blocks and Layouts initiative. This also allows for more crowd-sourcing opportunities with the Drupal community to work on the remainder of the initiative’s tasks.

The Web Services Landscape

This project is resourcing only a portion of the overall Web Services Initiative. It will provide foundational architecture that enables future Drupal 8 work, including improved performance via Edge Side Include (ESI) support, a flexible layout and block system, and better support for mobile devices.

The diagram below outlines both the work that is part of this proposal, and also summarizes remaining parts of the Web Services Initiative that are dependent upon this work.

Funding Proposal

In order to accelerate the velocity of the Web Services Initiative, the funding is being requested for the following key issues:

A) Drupal Kernel Patch (http://drupal.org/node/1463656). This restructures Drupal to use the Symfony request/response delivery flow as diagrammed above. This code will simplify and streamline the current complex system allowing developers a clear, consistent, and simple way to manage requests and responses in Drupal and enable communication between Drupal systems. This is an essential foundation to all future work in this initiative.
-- Development and discussion (3 weeks)

B) Routing System (http://groups.drupal.org/node/220269): This is required plumbing to allow for a Drupal request to return a themed HTML page to a desktop browser and an XML or JSON representation of the same resource to an iPhone app. It would also enable more robust Server-to-Server communication over a variety of REST-style protocols. This would happen automatically based on the incoming request. It provides underlying functionality to turn Drupal into a first-class REST-based web services framework.

Put more technically, this patch will extend Drupal’s current request routing system to include the ability to perform different logic depending on the HTTP method (GET, POST, PUT, etc.) acceptable mime type (text/html, application/json, etc.), and restrictions on the datatypes of parameters (int, string). This is critical, because it allows Drupal to reliably handle web services.

The Kernel Patch must be completed prior to work beginning on this system.
-- Development and discussion (4 weeks)

C) Canonical Format Discovery (http://groups.drupal.org/node/197583): Syndication of content from one Drupal site to another requires a common format for communication. A standard canonical export format opens up a wide range of possibilities for deployment, alternate storage, content syndication, and web services for Drupal-specific mobile apps. This discovery project will define this canonical format and the methods for implementing it within the Drupal project.
-- Discovery, discussion, and planning (3 weeks)

D) Sprint funding
Larry Garfield will be in attendance for Symfony Live in Paris (June 6-9). A sprint the previous weekend to discuss and plan a direction for the canonical export format would be valuable to form a clear direction for such work. Recommended attendees would be Larry, Wolfgang Ziegler (fago, Austria), Dick Olson (dixon_, Qatar), Yves Chedemois (Paris), and Damien Tournoud (Paris). That would likely be a 3-day sprint, and could be anywhere in the June 2-6 range, depending on travel schedules. Funding would include accommodation for Larry, Wolfgang and Dick; flights for Wolfgang and Dick.

Resources

Additional background reading, for the curious:

AttachmentSize
image00.png82.9 KB
image01.png52.73 KB
image02.png86.5 KB
LSD-WSCCI.pdf267.78 KB

Comments

Funding

fabpot's picture

SensioLabs can host the sprint and we can also fund some travel/hotel costs.

Thanks!

chrisstrahl's picture

Hey!

Moshe and Angie saw your post and are talking about incorporating this into the plans for the sprint.

Thanks!

Did you guys forget that you

cmcintosh's picture

Did you guys forget that you can write your own theme engine? This seems unnecessary.

If you think this is only

torgosPizza's picture

If you think this is only related to the theme engine, you need to read the provided links and discussion.

The theme system is a small

neclimdul's picture

The theme system is a small piece of the puzzle mostly for server side html building. Just a box on the edge of the diagrams here. That said, there's plenty of parallel work going on there to support a different set of users and use cases. There was a sprint last weekend that went quite well and we'll be hearing more about soon.

Plus this looks like it could

cmcintosh's picture

Plus this looks like it could break most existing things in drupal, i know you guys like the motto about breaking your code, but that tends to piss off customers and those who use the system and isnt seen as cute, but seen as a weakness.

Canonical Export Format Update

ethanw's picture

Quick note following up on the IRC chat about a week ago: research comparing options for export format is in progress, targeting a first version of the wiki page this weekend. That release will contain initial evaluations and outline a process for comparison (specific examples of specific encoded content features, etc.).

Awesome!

Crell's picture

I look forward to it!

So, if we want to chip in...

mherchel's picture

So, if a single person wants to donate a small amount, where can we do so?

I really think that this initiative (and it's cool sounding name) is really going to take Drupal 8 to the next level. Thanks for taking this on, Larry!

A really good question...

chrisstrahl's picture

I'm going to be working out a way that we can take some small donations associated with the project. Of course people are able to contribute at different levels, and the answer should be more than "Just give Acquia you're money and we'll take care of it." :-)

I'm going to be working something out with the DA or Acquia to provide the community transparency into how it would be spent. For now, if you want to hit me up on my contact form I can work with you individually to figure things out.

Timing is everything.

highermath's picture

While it doesn't hurt to ping the DA, being 6-7 weeks out makes it unlikely that you could get something in place when you need it -- a week ago.

Some brave soul with a good accountant needs to setup a ChipIn for this.

I have no doubt that we'll

AmyStephen's picture

I have no doubt that we'll look back at this decision and see it as a brilliant move for both projects.

or another half implemented

cmcintosh's picture

or another half implemented feature in drupal core.

An alternate proposal?

ethanw's picture

I'd be interested in seeing a fully-fleshed out proposal for how Drupal can implement a robust RESTful architecture (for starters) that does not suffer from the drawbacks of contrib approaches that Crell notes in this post and can scale up to the level of complexity and coverage that current state-of-the art clients (as in web clients, not project clients) demand. As Crell talked about in Denver, this work further sets the stage for improvements to Drupal's dependency management system and other sorely needed optimizations.

If you see a way that theme-level or contrib-module solutions could tackle these problems better, I, for one, would be very eager to hear more.

As a Drupal site builder I certainly share some of your concerns, but frankly, if all you have to offer are fairly unconstructive criticisms such as those above, I really don't see how they will contribute to a community project driven by a team that's putting a great deal of energy into some very difficult tasks.

Im not saying that this is

cmcintosh's picture

Im not saying that this is not a great idea, I would like to see it implemented out side of core first. We have had several great Ideas for d7 that we could not get in fully, ie Entities and it is unfortunate because they are so powerful and could go a long way to improve load time, etc.

With Drupal you have the standard Php template engine, but you can also write themes that use custom engines to process and display code. http://drupal.org/project/theme%20engines has a list of them. Could this not be done by writting a new template engine and supplying that with core? At least at that level it would not effect other things.

My other worry is that we build this and add it to code and it ends up adding more overhead and slows down drupal even more. D7 runs way too slow and consumes to many resources just for core functionality. Overlay is a good example of something that was meant as a good idea, but has caused other issues.

I want to be a bigger part of this process as well seeing as i have vested a lot of time and energy into drupal as a framework.

Maybe i am misunderstanding the full scope your looking to address, but I would rather not see Drupal as a big block of something trying to meet everyone's needs, and instead a more lean and quick Framework allowing people to plug into it to extend it.

Meaning moving things into a direction that would mean less things being loaded/scanned at every page load(only those things active and being used), etc.

The code is already looking

AmyStephen's picture

The code is already looking good. Check out those sandboxes. I see sanity coming into the core logic flow, making it easier to standardize events and triggers.

Today: http://groups.drupal.org/files/image02_0.png
Tomorrow: http://groups.drupal.org/files/image01_1.png

There's no doubt this will be disruptive. But, real innovation always is. Having said that, it is important to understand this is very low level architecture and something the average site builder will be blissfully unaware of. Should be fairly transparent to most.

I believe this has potential of outfitting a more modern technical foundation for Drupal that should help support 5 to 10 more years of community-driven innovation. Remarkable.

Real innovation is not always

cmcintosh's picture

Real innovation is not always disruptive and you should be more considerate of the average population who is looking for a STABLE platform to build on. Having a platform that is always changing fundamentally and breaking something that you had spent maybe 1k+ less than a year ago is not good. Businesses look at ROI and not just Innovation.

I would like to see this built into core in a way that it can be optionally used, and not forced on all sites.

I also dont see an issue with

cmcintosh's picture

I also dont see an issue with leaving this as 'middleware' its a better option allowing people to choose if their site needs it or not. If you can build it in that sort of way thing i think it would work out better.

Take it elsewhere

gdd's picture

This thread is about funding the proposal, not the technical aspects of the initiative. Technical and architectural concerns should be discussed in the issue queue and gdo threads around the initiative. I feel like these discussions should be kept there, and leave this thread open for discussing the funding possibilities and issues.

Would be glad to i didnt see

cmcintosh's picture

Would be glad to i didnt see any of those posts, I was only aware of this because i received an email about it. Would be glad to join the discussion and learn more about it. Not against it necessarily just concerned.

The initiative idea

Simmol's picture

I don't want be rood or something, but i suggest to take a look at the initiative discussions and the stuff talked so far, cause i think you not fully understand what is the idea.

If you don't hack Drupal Core to do you're work or you don't do work on the Drupal Core itself i don't see reason to worry so much about the change. It can only benefit the current scheme.

The change is fundamental in nature but will bring exactly what you refer as "lighter" Drupal in you're other comments.

And last it would be better if you join some discussion page or open new one instead of polluting this one witch is clearly for "Funding Proposal" stuff.

Hope there will be no hard feelings and have a nice day/night

D7 is stable and will be so

dalin's picture

D7 is stable and will be so for the next several years. If your website is around longer than that you'll either need to back-port security updates, or upgrade to D8/9. I'll leave the cost benefit/analysis up to you, but given that most security announcements only relate to vulnerabilities that can be exploited by someone with higher permissions, it is possible to extend the life of a D7 site well into the future.

Meanwhile the rest of us will be building new sites/applications/platforms/products/etc. that won't have been possible with the D7 architecture.

--


Dave Hansen-Lange
Director of Technical Strategy, Advomatic.com
Pronouns: he/him/his

This isn't the place to discuss this

chrisstrahl's picture

There have been plenty of threads discussing the basis, merits, alternatives, and plans for WSCCI. Almost all of these are linked inside of the WSCCI group. These discussions have been happening for months where dissenting opinions have been put forth and reviewed.

So please, this discussion is about the proposal to fund the initiative, not about discussing if WSCCI is a good idea.

I think that meeting was to

sumitone's picture

I think that meeting was to produced this proposal so got succeeded

Funding and Function

CowTownFarmBoy's picture

I do not have coin to contribute but have a word of advice. Innovation comes at a price and is not always accepted as the right thing until it has proven itself. The technology proposed has a high probability of working and should be attempted.

As one who has been working with and around web services for the better part of a decade and seeing the use, growth, and maturity of the architecture and the changes to our business and uses, I would say it is time to at least try and move the Drupal architecture forward.

As to funding, vote with your dollars and support. Those who want to move the platform forward will contribute and do so to the benefit of us all.

And to the merit of the technical proposal, it will be proven in the doing and use. But at least the Drupal platform and community deserve a shot at a better more up-to-date technology.