You may know RESTWS, a Drupal 7 module that exposes any entity as web service API. As soon as you enable it you can create, read, update and delete (CRUD) + query any entity type in JSON and XML. Take a look at the README.txt for further explanation.
I have not been involved in the WSCCI initiative so far, so my questions and assumptions might not be accurate (please correct me). I might be able to work on RESTWS for D8 and I'm looking into ways to fund my work right now. Here are some points that I'm considering about this effort:
1) I assume that we want a RESTWS-like module in Drupal core, as the goal of WSCCI is turning Drupal into a fully blown REST server. We now have property information about entities and we are working on JSON-LD serialization, so entities look like the perfect structures to expose out of the box as RESTful resources. Just as RESTWS does.
2) What would we call such a module in core? RESTWS was coined out of name space problems in drupal.org contrib. Just stick with it?
3) The ugly hook_menu_alter() of RESTWS needs to be converted to the new routing system.
4) The RESTWS info hooks for resource and format controllers should be turned into plugins.
5) Ideally serialization is not a concern of D8 RESTWS. Just use the serialization module (will there be one?) as black box to convert entities. That might not work for querying, where we need to do paging and add hyper links to navigate for clients. Anyway, I consider querying out of scope for D8 core (RESTWS uses EFQ to retrieve for example all nodes of type story as JSON).
6) What is the overlap with the plans for Deploy module? Could D8 RESTWS be the foundation for Deploy?
7) Currently we use HTTP PUT for updating entities in RESTWS. According to the HTTP spec PUT should be idempotent, which is strictly speaking not the case for most updates in Drupal. Example: a module might implement hook_node_update() and change the node when it is updated via the web service. The node in the system is different to the transmitted node now, which contradicts idempotent. So we could use POST instead, but we would lose the clear HTTP mapping that we have in RESTWS right now: create = POST, read = GET, update = PUT, delete = DELETE. POST could then mean create or update, and we have to figure out what is actually meant in another way.
8) Authorization and authentication: RESTWS D7 completely relies on the user/role/permission system, a client has to authenticate via cookies or with restws_basic_auth.module via HTTP basic authentication headers. This is elegant because RESTWS can completely ignore authentication and permissions. Just create a user account manually and assign permissions to entities/resources that should be accessible. I suggest to do the same for D8.
9) URIs: RESTWS uses the canonical path /<entity_type>/<entity_id>, e.g /node/5, /user/78. Problem: that might not match always. Example: /taxonomy_term/5, which is used as /taxonomy/term/5 for HTML responses. According to REST it is important to have resources at one canonical path only. See also: Give all entities their own URI: http://drupal.org/node/1803586 .
My plan would be to provide a minimum viable product first, where we implement only deleting entities (which is obviously the simplest operation as no format/serialization is involved). This should cover the routing and the general controller flow, including test cases of course.
I would like to get some feedback if the general approach of RESTWS is the thing that we envision for web service entity CRUD in D8 core.
Comments
Fingers crossed for funding!
Fingers crossed for funding! This would be a great addition to WSCCI.
I'm currently exploring this, and spent yesterday working with Symfony's Serializer component. I'm thinking that we can rely on Serializer for most of the work.
The way I understand how JSON-LD would integrate with the new routing system, a route would be defined for the format, and mapped to a controller which would be specific to that format. In the last WSCCI meeting, we discussed this issue (I can send you the transcript if you like).
Instead, I would prefer to have format agnostic routes and controllers (at least for non-HTML serializations), and think that we should be able to support that by toggling between different Normalizers and Encoders (parts of the Serializer component). I still have to test this.
It sounds like Rest WS would provide the format agnostic routes and controllers, which I would be happy to see.
Related to this, two issues have suggested standardizing our URI template for entities [1] [2].
.
I am a little worried that we won't be able to use the json-ld implementation from github with Serializer.
Sprint in Chicago? Ohio is too late being at the feature freeze deadline itself. :(
Which JSON-LD implementation
Which JSON-LD implementation on GitHub are you referring to? There are at least 2 PHP implementations of the spec on github and I believe there are a number of JS ones there too.
Also, what are your specific concerns regarding incompatibility between the two? The JSON-LD spec currently focuses on expansion/compaction and conversion to RDF. For that, you assume that you have JSON-LD already and are going between the compact and expanded form. We don't plan to support conversion from RDF since we are using a part of the JSON-LD data model that is not supported in the RDF data model. We may need to include compaction/expansion in the Denormalizer, but I think we could use one of the JSON-LD libraries from github and depend upon that in our Denormalizer.
I will be sprinting at BADcamp, I had not planned to go to Chicago since that's the weekend after. I can sprint virtually, though, or would be up for going if there is funding support.
I must have misremembered
I must have misremembered some things from Munich. I had thought that we did not want to write our own implementation and I must not have found the correct library because that one (don't remember which) didn't look like it would fit in with the Serializer/Normalizer stuff. Sorry for the confusion.
How do you intend to use Symfony's Serializer?
Lin,
how do you plan to use Symfony's Serializer component with JSON-LD? Are you going to use the JSON serializer and just add a context afterwards?
What about deserialization of requests? I played a bit with the form framework to deserialize a JSON-LD request and handle it just as a request from an ordinary HTML form would have been handled. The structure of the request doesn't matter at all as I used an in-memory representation of the graph contained in the request. The only requirement currently is that the entity in the graph is typed (to be able to find the right one).
Markus Lanthaler
@markuslanthaler
My plan was to handle adding
My plan was to handle adding the context (referenced by URI) in a Normalizer. That Normalizer would only be used if 1) it is a supported object (a subclass of Entity in our system) and 2) if the requested format was JSON-LD.
I'd be happy to hear feedback on this approach.
Regarding the form framework, we don't plan to use that part of Symfony in D8. I haven't worked through deserialization as much yet, but my plan would be to write a custom Denormalizer. Klausi's controller would then call denormalize for POST and PUT requests to extract the Drupal entity from that.
Sounds like a straightforward
Sounds like a straightforward approach that makes a lot of sense IMHO. Is the context constructed automatically by minting IRIs for all Entity classes and their properties?
If so, deserialization would also be quite simple.. look at the entity's @type to find out which class has to be used and then map all properties from the request to that instance.
Let me know if you have questions or need help. I don't have much time at the moment but I'm very interested in this since I'm working on something very similar in my spare time.
Markus Lanthaler
@markuslanthaler
You are correct about the
You are correct about the context. In Drupal we have entity types, like Node, and entity subtypes (confusingly called bundles). An Article is a subtype of Node. There will be a context per subtype.
Thanks for the offer of help, I will be sure to let you know if I have questions, and feel free to chime in with comments/feedback whenever :)
Yeah, this is the approach I
Yeah, this is the approach I was working on in the patch in your sandbox though I was just trying to get JSON itself to work.
Yay!
Thanks for this post, klausi. I pretty much agree with all of it.
1, 3, 6, 9) yes
Does idempotence require that what is saved be identical to what is transmitted, or only that retransmitting the original transmission not cause another change in the system? In any case, we definitely should not violate whatever the requirement actually is, so we may need to have a way to let modules that hook into an entity update in a way that breaks idempotence to also have a way to flag that and cause the PUT to be denied. Which would also mean needing to support updates via POST as a fallback.
Oh, I did not remeber
Oh, I did not remeber idempotence correctly, you are right. Anyway, updating a node multiple times might not result in the same system state, as other modules can have arbitrary side effects at an update.
POST could be a solution, but then we lose the nice mapping of exactly one HTTP operation to exactly one CRUD operation, which I find very appealing.
Correctness.
I think we should research if our update maps correctly to PUT, if not - don't use it and go with POST. Drupal should not be famous for a not really RESTful "REST-API".
Afaik, PUT requires that you exactly put the posted content to the resource. Besides node updates (what maybe does not bother), this won't work on create as we don't know the ID and so the URI yet. So, PUT won't work then, so it's usefulness would be limitted.
Next, we have field access would the web service will have to respect. But that means that we have to support passing filtered or differently said partly entity representations, what clearly violates PUT as to my understanding of it. Because of that, I think it's a no go.
That's simple, as it would clearly map to an entity save operation.
There are two things I'd like
There are two things I'd like to see in addition to the above, which I think is a very sane set of implementation goals.
1) Pluggable authentication. This way we aren't alienating people who want to use oAuth or whatever other schemes come up in the next five years. We don't have to provide any other implementations than core users and sessions, but the ability to make this pluggable is I think a key feature. Services has this now and as far a I know it works well (I haven't been involved in quite a while.)
2) Expandability aka if I run Flag module, and Flag for whatever reason decides not to use entities, I want the ability to hook into this and expose my data as a REST api as well.
Versionability as a side effect of 2 would be realllly nice.
Non-entities
There's two aspects to this:
1) Exposing things or lists of things that are not entities (e.g., the list of enabled modules). I don't think it's a requirement that this have any implementation/consideration in core. A contrib module could always add additional routes to handle those, even if whatever restws implementation in core does nothing itself to provide extensibility for that.
2) Exposing information on an entity that is not a built-in property of the entity (e.g., $node->status) or a field. Flag could well be an example of such a non-field (at least in D7) entity extension. However, the new (and still being refined) Typed Data / Property / Field API is specifically designed to allow all entity data extensions to be exposed in a common way so that a serialization engine can handle all of them with no need for modules like Flag to require special serialization add-ons. This is key to allowing D8 contrib space to support multiple serialization formats cleanly.
1) As Moshe already said I
1) As Moshe already said I think authentication is out of scope for implementing RESTWS-style CRUD. There is not enough time left until December 1st.
2) The property API should be the uniform metadata interface that we use to expose stuff. In RESTWS D7 we use the concept of resource controllers that can provide property information for non-entity data structures.
Versionability is not possible with a generic solution. I can add or remove a field on a node and the "schema" changes. I don't see a way how we should automatically map old incoming requests to an outdated version of a node type. So this is clearly out of scope for this effort as well.
7) When you PUT the same
7) When you PUT the same values multiple times, what changes? The Last Updated timestamp changes, and possibly a duplicate revision gets created. That strikes me as completely reasonable, and close enough to idempotent. I'm not opposed to re-using POST, but I think PUT would be fine too.
All your suggestions and assumptions look sane to me.
@heyrocker - I think pluggable auth is strictly out of scope here, though it would be awesome if someone worked on that. I guess Services project would be a reasonable place to start.
Welcome aboard!
Hi, Klausi! Welcome aboard.
In no particular order:
Yes, ideally we want
GET /node/5 Accept: application/jsonto "just work". In theory we won't even need an alter for it, although that's still a work in progress to support that. You'd just define an additional route with the same path and Bob's your uncle.For access control, I don't see why it would be any different than any other route. If you want to allow authentication via OAuth, you flag the route to use OAuth and then add a request listener that checks OAuth status. Or, alternatively, you implement a Condition plugin that ties into the route authentication system that we're still working on: http://drupal.org/node/1793520
I need to consult with some experts on just how idempotent idempotent has to be in order to be idempotent. :-) I know that incidental things like logging do not violate idempotence. OTOH, I don't know if enabling revisions on a node type would inherently make it non-idempotent in all cases. We need more research here.
IMO, RESTWS as a module doesn't need to continue. Rather, its functionality just folds down into the base system directly without the need for additional libraries for the general case. Format negotiation is handled by HTTP. Authentication by route authentication in general. URIs are standardized across mime types, and between Drupal sites. Etc. For that reason, I'd love to have you working with us in core so that we can get to that point. :-)
Well, for me the question is
Well, for me the question is now: Should we have a core module that explicitly enables a web service entity API where third parties can interact with JSON-LD or similar? Or do we just provide the foundation to make that happen and a module like RESTWS is still needed in contrib? I assume that there are a lot of Drupal sites out there that do not care about or do not want to expose JSON on a node path. So by default we should not add any routes that listen for such requests, a dedicated module should enable them.
Access control: Sounds like we can off-load that to the routing system with RESTWS-independent mechanisms. I like that!
I think RESTWS should continue as rest.module (name is work in progress) in Drupal core. It should wire up serialization with routing. Users should be able to simply enable a web API for an entity type. RESTWS currently just exposes any entity type as soon as it is enabled, I don't think that is always a good thing nor necessary. We should avoid routes that are not needed, which increases performance and improves security.
This makes sense about access
This makes sense about access control. It's not as seamless but as long as we have options then I'm totally cool with it. I'm honestly really excited to see this work going forward.
My goal is that no Contrib is
My goal is that no Contrib is needed to get full CRUD web services. I agree with a rest.module in core that exposes these routes. Exposing certain content types makes sense.
Yes. I agree that a simple
Yes. I agree that a simple module that just provides the basic UI for enableing/disable web service support per entity type and content type would be neat. Although, maybe the media type could be also configured as page manager variant and the ws-module could just add in the page and/or the variants? Just an idea...
Great initiative. I believe
Great initiative. I believe that this should end up with something like we have XMLRPC in the core, i.e. something lightweight to set up. But it looks like we will still need contrib module to provide endpoints, authentication, custom resources etc. So possibly services module will still live in some form for drupal 8.
Also we haven't touched part to optimize loading of the code for the REST calls, so we for example won't load theme during the request.
HATEOAS
I have recently been researching REST APIs over the last few weeks. I'm working on a similar project, but it's the reverse of this. I'm creating an Entity Storage Controller that uses an external REST Server to handle all of the Entity CRUD for integration with an external system’s data. This exposes external data as native Drupal Entities. I'm also working on creating a REST API Server for that external system which Drupal will use.
1) You should also look at JSON-Schema:
- http://json-schema.org/
2) Call it what it is, "restapi". This is an unsupported module consider using it:
- http://drupal.org/project/restapi
7) There is nothing non-idempotent about using PUT where some module calls hook_node_update(), as long as you get the same result if you PUT that resource again. If server side calls to hook_node_update() produces the same result when the same resource is PUT multiple times in a row, then it would be idempotent. If something on the REST server modifies a resource when it is PUT, that doesn't mean it's not idempotent. Idempotent simply requires that if PUT is called 5 times in a row with the same resource body, you will still have the same state each time. Making a PUT over and over should not create a new resource each time it's called. I believe thinking that the REST server cannot modify a resource after a PUT is the wrong understanding of idempotent. POST is for creating, PUT is for updating. If another PUT would change what has happened, you should have a way to fail gracefully and return an error. This error would say that the resource you are PUTting is stale and no longer valid. It should be reloaded, modified and then PUT again.
To fago's point about "we have to support passing filtered or differently said partly entity representations" that should use PATCH, not PUT. PATCH is for partial resource updates.
9) A RESTful Server does not need canonical paths or a single path for each resource. The only important path is the root URI for the REST API service. All other paths should be discoverable by the client at run time. Your resource paths could changes each day if you are following true REST Style in terms of HATEOAS. As long as the root URI for the API does not change and the hypermedia it returns contains the URI links to the resources, it doesn't matter what the path is or that you can get a similar resource in different formats from different path structures. Obviously, you should try to avoid the exact same response/format/resource being in multiple locations, but I see nothing wrong with “/taxonomy_term/5, which is used as /taxonomy/term/5 for HTML responses” under REST with HATEOAS. As long as something links to both URIs and there’s enough metadata for the REST Client to discover what it needs, path structure doesn't really matter.
One key concept to REST I've been exploring is HATEOAS. RESTWS doesn't really conform to HATEOAS as well as it should. I would recommend you take a look a this video which presents a good example about how the REST Server should provide the hypermedia responses in a HATEOAS way that allows the REST Client to dynamically adapt to changes:
- http://vimeo.com/20781278
I have also been looking into Google's API Discovery Service as an approach to HATEOAS.
- https://developers.google.com/discovery/
- http://www.youtube.com/watch?v=lQbT1NrxpUo
PUT redirect
Newbie here, just to note that you could redirect a PUT request with a 301. From http://www.w3.org/Protocols/rfc2616/rfc2616-sec9.html#sec9.6:
(disclaimer: this is personal preference) I think idempotency refers to the cachability of request-response pair by the middle-man (proxy).
Let say we have this:
The proxy can say, "hey I've seen this 2 and 3, the server dude didn't say we couldn't cache what he said, so lets drop those two, give the client what he gave us earlier."
If for instance we have this:
The sequence is no longer idempotent and the 3rd request cannot be dropped by the proxy.
If what you want is to update the URI's modification time, you could use POST (because it is not cached) with the URI as the container, and the modified Field as the item we want to alter. Because PUT sucks at this.