The push towards JSON-LD support in Drupal

As Larry talked about in his WSCCI Web Services Format Sprint Report, we want Drupal 8 to be able to serialize data "for consumption by remote programs (either other Drupal sites as in the case of content staging, other non-Drupal sites, client-side applications, or mobile apps)".

During the Web Services format sprint, JSON-LD was picked as the most promising candidate for the primary supported web data format. This means that though it will be possible to support other formats in contributed modules, JSON-LD will be the format that we will focus on most during the core development process.

A lot of work has been done to ensure that we have the proper foundation in place for Web Services, from adopting many Symfony components, to revamping our Entity API, to refactoring routing. Now that a good portion of the foundational work is done (or done enough), the next question on the horizon is how we're going to leverage that work to provide the serialization.

Progress

In parallel with the routing and Entity API issues, we've been working to ensure that the JSON-LD output supports our different use cases.

We've been defining our requirements [1] [2] [3] and working with the JSON-LD Working Group to request changes to the spec [1] [2]. We've come up with a basic plan for what a JSON-LD representation of an entity will look like. And we've started putting together a roadmap for adding JSON-LD support.

But the devil is in the details

JSON-LD is more flexible than many other formats. It provides an infinitely extensible mechanism for registering properties and a very general data model, which can be used in a number of different ways. As the spec says:

The syntax must be able to express directed graphs, which have been proven to be able to express almost every real world data model.

This flexibility means that it can be used to fulfill many use cases. But it also means that simply using JSON-LD doesn't guarentee compatibility with JSON-LD consuming tools. The data model and vocabularies used in the JSON-LD have to be understood by the consumer you are targeting.

We are trying to ensure compatibility with a wide range of tools, to support a wide range of use cases. If we are to achieve this, we're going to need all eyes on deck. We're going to need people who are familiar with the requirements of different use cases to keep an eye on the serialization as it progresses and ask questions as needed. We're going to need those people to give clear feedback if a change negatively impacts their use case, and to be open to brainstorming solutions.

There will likely be points where we need to prioritize some use cases above others, and I will be coming to this group for discussion at those points. Hopefully we can keep those discussions focused, because we don't have much time if we want to support this in core, which brings us to...

With 9 weeks, how are we going to get this done?

There's approximately 9 weeks until feature freeze. Yikes!

We're working with spec that's under development, and building on top of code that's currently in-progress. We will have to prioritize.

Step 1: Responding to requests with JSON

The first step will to respond to JSON/JSON-LD requests for entities based on Accept header and HTTP method. A significant challenge will be defining the full entity serialization. We want to align our entity data model to consumer expectations while still providing the expressiveness we need for things like content staging.

In this first step, we will simply respond with JSON. The JSON object's structure will be based on JSON-LD, using the same syntax for node definitions, node references, and the language map hierarchical structure, but will not include the '@context' attribute (which gives JSON-LD much of its richness).

Even if we just get that work done, we'll have taken a big step forward.

Fortunately, Acquia has offered their support and will partially fund my development time for this phase. Thank you to Stéphane Corlosquet for initiating and coordinating the funding proposal. Stéphane will also be working on content negotiation as part of this work.

Step 2: Adding LD to JSON

The subtitle of the JSON-LD spec is "A Context-based JSON Serialization for Linking Data". The context provides additional information for processing, such as language, datatypes, and universal identifiers for properties.

In order to have JSON-LD support (as opposed to regular JSON), we need context, and would need to resolve at least the first couple of issues listed under Step 2 in the roadmap.

Step 3: Provide additional features for consumers

Once we are returning objects that can be processed by JSON-LD processors, we can use other REST and Linked Data conventions to help consumers make better use of the data. Which conventions we would want to use have not yet been clearly defined.

Follow the discussion

As I mentioned, I will be posting here at some of the key decision points.

If you're going to be using JSON-LD in Drupal 8, you will also want to watch the issues (listed in [META] JSON-LD support). I'm also available in #drupal-wscci if you want to talk about use cases or have other ideas around this work.