Proposed Goal: Many Implementations of a Common Demo

We encourage users to post events happening in the community to the community events group on https://www.drupal.org.
joshk's picture

My biggest concern is that this group ends up bikeshedding and/or attempting to boil the ocean and solve a million theoretical problems. My biggest hope is that we deliver a killer app for front-end developers looking to build their implementation. If we can do that, it will force us to do a lot of other smart things, and enable millions of others to put their innovation "on top" of ours.

To that end, I'd like to propose a goal that's pragmatic, provides value, and will help keep the discussion and work focused.

We should create several JS client implementations that can accomplish a common demo.

What I imagine would be similar to how TodoMVC functions in the world of JS Frameworks. Not for nothing, but the WordPress API group is doing this too, and it's helping them get traction.

The basic demo would be something like:

  1. Page/Article CRUD
  2. Extend the data model for content
  3. User auth/operations
  4. Drupal admin operations?

We should be focusing on making it really easy and really elegant to do these things for front-end developers who don't know Drupal from a hole in the ground.

We can start by making sure we can do it ourselves. With these demos "in the wild" we can attract new contributions and help jumpstart a new generation of development and innovation.


UPDATE: See comments below. It occurs to me that we should not limit the implementations to JS. Having a Guzzle implementation, a Python implementation, a Ruby implementation, heck even a CURL implementation would all be excellent and helpful.

Comments

A woefully incomplete example

joshk's picture

Here's what I was able to whip up this afternoon for Angular:

http://dev-headless-drupal-8.gotpantheon.com/

This is build on Alpha 12 so not the latest; hopefully we can address some of the shortcomings as the Beta comes out!

Where I feel blocked:

  • REST module doesn't deliver "hydrated" node objects or seem to provide any way to access an extended data structure. All my nodes have images, but I don't know anything about them in the REST response. Can't do anything fun with the data model.
  • Cookie authentication is under-documented and I can't make anything work. In fact, if I hit any REST callback with my session cookie, it terminates the session. Adding the X-CSRF-Token header didn't fix it.
  • Assuming I can get auth working, it's unclear what the REST targets for create/update/delete operations would be.

Where I feel annoyed

  • In general the documentation is super lacking. Part of the objective here is to help fix this, but it's still not a lot of fun to have to hunt around for a long time just getting the basics set up.
  • Everything that comes out is nested in arrays which makes my HTML uglier than it needs to be and complicates any data logic. For instance I need to reference {{ node.title[0].value }} instead of just {{ node.title }}.
  • I understand the systems reasoning for having the REST module live separately from Views rest display, but it feels strange that I go down one path to read a single node, vs get a list of nodes.

More to come!

Reference Implementation

eatings's picture

I agree that we need a reference implementation of some kind to benchmark against, or at least one to test assumptions against real-world implications that don't arise until we try to just do it.

We need an engine

skriptble's picture

I agree with you Josh.

  • Drupal's auth is lacking
    • cookie based auth isn't appropriate for an API
    • Basic auth doesn't provide enough flexibility.
    • We need OAuth or develop an authentication scheme using something like HMAC or JSON Web Tokens.
      • If we do go down the oAuth road we should ensure our implementation contains a way to separate the resource and authorization servers.
  • HAL is a limited format: it has no way of displaying actions, which is crucial for hypermedia APIs, but okay for REST APIs.
    • If we stick with HAL we should probably document that it's a read only hypermedia format
    • We should probably offer another hypermedia format such as Siren or Hydra
  • We should put an abstraction layer between our Drupaly Content and the outside world. Just like we have an interface to choose which fields, we should have an interface to translate from Drupal to an abstracted media type

A great way to solve this would be to package it up into an engine that can not only map entities to hypermedia formats, but also provide a way to add additional information such as actions, relations, and authentication.It can also handle submissions and handle translating back to Drupaly Content. As a bonus if we can add some machine readable profiles (like ALPS), we could have a fully functional Hypermedia API system. At the least though, we should translate the content and get a strong authentication method.

We need clients

joshk's picture

Hey Kris,

I hear all your points, but I feel like this is the kind of discussion — loading up on backend requirements — that I'm trying to steer away from, at least on this thread. My point is that we should put work into building clients, and let that guide further work.

For instance, if Cookie auth worked well it would be fine (superior even) for a "headless front-end" implementation in various Javascript frameworks. Since the user-agent is the operating system for the client, it's quite natural: Login via XMLHttpRequest (or /user/login), get cookie, get X-CRF-Token, keep on rockin'.

Similarly, before we start coming up with more API requirements (for other systems consumption) we should make sure we have a client use-case in mind — if only an example! — so that our decisions are informed by the actual needs of someone implementing said API.

data abstraction layer

scottrigby's picture

@skriptable Re:

We should put an abstraction layer between our Drupaly Content and the outside world. Just like we have an interface to choose which fields, we should have an interface to translate from Drupal to an abstracted media type

This comment describes the mapper I was talking about at the NYCCamp planning meeting. If there isn't already something like this for D8, adding that could help this issue.

Serialization API

skriptble's picture

Agreed. I think we can implement this using the Serialization API in D8.

Great to see that!

andypost's picture

Suppose for now demo could use basic auth and then contrib could implement others.

It would be great to share the code in some sandbox/github also I think having install profile for the demo is really needed.

PS: I'd like to help here but not idea how. Wscci initiative lacks of feedback and attention from real usage

Github Coming

joshk's picture

I will have this up in a open repo this week. Demoing it tomorrow at SFDUG

SFDUG

RobKoberg's picture

Hmmm.. might have to come down (I am in Marin). When/where?

I will come. If you like, I

RobKoberg's picture

I will come. If you like, I have a D7 app that is a couple of months in I can show you. (no end time showing for the meetup, might need to leave early to catch the ferry)

Can't make it. Hope to see the video, though. Love to hear about what you all are up to!

Great idea. Glad to see

RobKoberg's picture

Great idea. Glad to see movement away from the bikeshedding. Have you seen https://github.com/angular-app/angular-app?

"The idea is to demonstrate how to write a typical, non-trivial CRUD application using AngularJS. To showcase AngularJS in its most advantageous environment we've set out to write a simplified project management tool supporting teams using the SCRUM methodology. The sample application tries to show best practices when it comes to: folders structure, using modules, testing, communicating with a REST back-end, organizing navigation, addressing security concerns (authentication / authorization)."

The thing to take away from the example best practices app is to build in a modular way (seems familiar, huh?). Instead of having a folder for controllers, one for services, one for directives etc, group code together that can be reused. I will start a thread for angular best practices, tips, and tricks.

Make Drupal a Backend for Angular App?

joshk's picture

I was not aware of that. Thanks for the link.

Do you think making Drupal a solid/easy backend for angular-app could be one of the goals?

It would be difficult

RobKoberg's picture

It would need to somehow transform the array hell to an easily consumable json format. I doubt that will happen in the core. I have been using the alters and menu items to produce the response to the client. For example:

function mymodule_node_view_alter(&$build) {
  switch($build['#bundle']) {
  case 'online_book':
    $node_wrap = entity_metadata_wrapper('node', $build['#node']);

    $response = array();
    $response['build'] = $build;
    $response['nid'] = $node_wrap->getIdentifier();
    $response['pageTitle'] = $node_wrap->title->value();
    $response['slug'] = $node_wrap->field_book_slug->value();
    $response['body'] = $node_wrap->body->value->value();
    $response['cover'] = render($build['field_book_cover']);
    $response['authors'] = $node_wrap->field_authors->value();
    $flag_counts = flag_get_counts('node', $response['nid']);
    $response['likes'] = $flag_counts['like'];
    $response['categories'] = array();
    foreach($build['field_book_genre']['#items'] as $index=>$item) {
      $response['categories'][] = array(
        'tid' => $item['tid'],
        'name' => $item['taxonomy_term']->name,
      );
    }

    drupal_json_output($response);
    drupal_exit();
  }
}

Same thing for forms where I set the values to provide a more client side friendly (and much lighter) model. I tried just taking the form array as given by drupal, but it seemed easier to just provide the response. entity_metadata_wrapper has been extremely helpful for this.

Plus there are weird things like image handling that will be difficult. In my case I am using an AWS S3 and the drupal s3fs module. I could not just get the s3 URL, so I had to render the image element. There is more iinfo about that in this issue: https://www.drupal.org/node/2322943

btw, $response['build'] =

RobKoberg's picture

btw, $response['build'] = $build; is in there for debugging. Kind of like a client side dpm(). That would be removed once everything is solid.

For D7, I worked on a mapping

scottrigby's picture

For D7, I worked on a mapping layer for RESTful Web Services - letting you map Drupal-structured data to a set schema (eg schema.org, etc):

https://github.com/scottrigby/restws_schema

This was helpful for allowing our AngularJS to use the same templates generated by an in-browser frontend prototyping tool of our choice (whether https://github.com/north/generator-style-prototype, http://patternlab.io, etc).

Also - yeah - we have been working on a step-by-step guide (using Yeoman etc) to building an Angular app for this backend... I agree with Moshe this is an important conversation to get going - thanks Josh! Looking forward to what comes out of it

D7 Demos Welcome

joshk's picture

I think getting knowledge out there about how people do this already in D7 is just as valuable. I'd love to have those examples in any showcase as well.

Can't we talk to Drupal without the mapping?

scottnath's picture

I worked with Scott, providing the Angular App. What I built was created using aaronallport's angular require generator, which is an Angular Yeoman generator that uses RequireJS to load scripts. This gist: Angular service to talk to Drupal, shows how we used Restangular to make REST calls and the code follows the file-created format of angular-require.

Anywho...what I found, after Scott Rigby set up his Drupal + Mapping POC, was that it was fairly simple to get data out of Drupal and inject it into the Angular App.

Right now, I'm working on creating an angular-app/dev environment combo - similar to a number of generators out there - but with the various needs we'll have for development and exporting for Production.

Scott, your mapping layer is fantastic for it's normalization, but aren't we able to make Drupal REST calls without it? Wasn't there a single module which you turned on that opened up Drupal's REST and made it a service? Did that open up the images as well? I remember you had galleries in your data...

Yes and no

scottrigby's picture

@scottnath restws_schema just helps restws serve an arbitrary schema rather than Drupal's Entity schema. There are other ways to provide REST via Drupal 7, but I chose to build it on top of restws because it is similar to REST in D8. I'm still interested in how https://github.com/Gizra/restful differs from restws (restful was not available when I made restws_schema).

The concept of RESTful is

amitaibu's picture

The concept of RESTful is here https://github.com/Gizra/restful/blob/7.x-1.x/README.md#concept

In short: versionable api, endpoints in expected places (e.g. api/v1/articles), but most important -- each resource is declared explicitly - so no Drupalism is exposed.

Checkout the blog posts and example modules.

Oh great

scottrigby's picture

@amitai From the concept description this sounds very similar to what we've been working on - I'll reach out again about looking at these together - thanks!

Sure, my offer for a Skype

amitaibu's picture

Sure, my offer for a Skype still stands :)

Have a look at the code, and examples -- you'll see that RESTful is taking a very different approach than RetWs.

This would be great for JS devs

RobKoberg's picture

I really like the idea of this (restws_schema) module. A JS dev could come in and setup the mapping through a UI. However, I went to install the module(s) but don't see them working (at least the UI one). Is the project just in its early stages or should it be working? I added a couple of issues to the github project with details: https://github.com/scottrigby/restws_schema/issues

Developers need to provide a schema

scottrigby's picture

I realize the README could be more explicit on this point. I'll update that soon.

In the meantime, I replied to your github issue, but here is the response in case it's helpful to anyone else:

@rkoberg The module requires two variables to be set in order to be useful. The first is restws_schema, which defines your custom schema - and which the restws_schema_ui module uses to allow mapping via the UI. Here's an example of that variable (with some more explanation):
https://gist.github.com/scottrigby/164bfec14dcb80f7f6bf

This mapping is the easiest way to create the second variable restws_schema_map, which provides the mapping layer on top of restws and allows your defined schema to be served. Alternately, you can just deploy this variable in code and the restws_schema_ui module in that case is only valuable for reporting (but it is still optional). Here's an example of that variable (also with more explanation):
https://gist.github.com/scottrigby/a8aa0186d2465b22702a

Thanks for getting this

moshe weitzman's picture

Thanks for getting this conversation going. And kudos for following through with some code.

  1. Imagefields don't yet participate in rest responses but we are not that far away. See https://www.drupal.org/node/1927648. Please elaborate on anything else you are hoping for in these responses.
  2. We got Basic Auth into core. OAuth is in Contrib

It took many months of hard work to get the rest responses into core. We need another wave of conributors. This effort could be a great catalyst. Go team!

RobKoberg's picture

I see the link about image fields is about POST/PATCH, but what about a GET? I would like to see the response as either a simple object (url, width, height, alt, title) or even just the correct current requested display's URL to the image.

Also, how will this work when using a CDN? In D7 there seems to be a problem getting the correct URL. See https://www.drupal.org/node/2322943

Thanks for the link!

joshk's picture

The Image issue seems mostly to be about POSTing new data. I suppose there's additional work to be done to get GET requests working?

I'll start keeping track of these issues as a way to try and drive more activity. Adding this to the body of my post. :)

RESTFul 8

amitaibu's picture

fyi, RESTful is being ported now to D8...

Sweet!

joshk's picture

I'll have to work on that for another demo. :)

Here's a good first step:

amitaibu's picture

Here's a good first step: Todo app with RESTful backend -- http://www.gizra.com/content/todo-restful-backend/

I agree with you that we need

shuneo's picture

I agree with you that we need a reference implementation of some kind to benchmark against, or at least one to test assumptions against real-world implications that don't arise until we try to just do it.

My name is insomniaku

Headless Drupal

Group organizers

Group notifications

This group offers an RSS feed. Or subscribe to these personalized, sitewide feeds: