Abstracted Web Services API for Drupal 7

We encourage users to post events happening in the community to the community events group on https://www.drupal.org.
RobLoach's picture

This is the beginnings of a design specification proposal for an abstracted API for implementing web services within Drupal 7. This allows Drupal to act as both a server and client for web services in any protocol medium (XML-RPC, RDF, RSS, JSON, etc). It is the combined efforts from a number of people at Druplicon including Marc Ingram, Scott Nelson, Nedjo Rogers, Dmitri Gaskin, etc.

Services API

The Services API allows you to host local servers in different protocols, as well as make web service calls to external servers. In all four of these functions, if you pass a $source and $protocol, the correct hook_client function will be called to make these external calls. These functions reside in services.inc.

  • services_get_server($server = NULL)
    • Returns all servers available to the local system (XML-RPC, JSON, RDF, etc). If a $server is given, it would return all information regarding that server. Only local functionality here.
  • services_get_services($source = NULL, $protocol = NULL)
    • Calls hook_services and retrieves all services within the system (external on the given protocol, or local)
  • services_get_service($method, $source = NULL, $protocol = NULL);
    • Returns all information regarding the given service (either externally or locally)
  • services_call_service($method, $parameters = array(), $source = NULL, $protocol = NULL)
    • Uses services_get_service to retrieve information about the method, and then calls its method (either externally or locally)

Hooks

These are the hooks that become available by the Services API:

  • hook_services
    • An array of all services with the key being the method name of the service (node.load, node.save, etc), and then the following properties:
      • #description
        • A short description of what the service does
      • #access
        • The permission the user is required to call the service
      • #callback
        • The callback function when the service is called
      • #type
        • The return type of the service call (array, string, number)
  • hook_servers
    • An array of all servers with the key being the short name of the server (rdf, xmlrpc, amfphp, etc), with the following properties:
      • #description
        • A short description of the server
      • #title
        • The title of the server (RDF, XML-RPC, JSON, etc)

Modules

The following are the new modules that are implemented:

  • Services module
    • Implements the administration interface to test services
    • services_menu for entry points to servers (example.com/services/%server)
    • services_perm for a user permission to "administer services"
  • xmlrpc_services Module
    • Implements xmlrpc_servers function to implement XML-RPC server
    • Implements xmlrpc_client to query a remote XML-RPC server
  • json_services Module
    • Implement json_servers function to imlement JSON server
    • Implements json_client to query a remote JSON server?
  • rdf_services Module
    • Implement rdf_servers function to implement RDF server
    • Implement rdf_client function to query a remote RDF server

If we wanted, all of these modules could be implemented into the Services module through the same hook (services_servers). This would allow us to only add one module to Drupal core as opposed to four. Nedjo brought up some good comments about why separating them is a good thing.

Hook Implementations

Implement hook_services for certain modules

  • node_services
  • comment_services
  • taxonomy_services
  • menu_services
  • search_services
  • system_services
  • user_services
  • BlogAPI would implement hook_services for the MovableType compatibility, when upgrading, enable Services and XML-RPC if BlogAPI is enabled

Aggregator?

  • Is a client for an RSS server?

And an RSS Module?

  • Is a server for RSS clients?

Comments

At the code sprint myself

marcingy's picture

At the code sprint myself and Scott spent sometime looking at the outstanding faults for the services module. All of the issue we believe need to be tackled have been assigned to my user name. It would helpful if people code review what is there and raise any additional items that they believe are currently missing. Please note services in core will only support functions carried out by core e.g at the moment there will not be a views services instead it will remain as contrib.

The current intent is to apply all of these fixes to version 5 of services, port them forward to 6 and then start work on refactoring the code to get it into core.

We discussed what servers should be included in core came to the view that it should only be xml-rpc.

What are peoples thoughts on this approach?

Marc

Sounds good to me

gdd's picture

I assume the best bet for non-core services is to submit patches and hope they get integrated into the modules themselves. Are you going to take this path for 5/6 as well, or leave views in there for now?

I plan to try and review some patches this week, hopefully I can you get some feedback. A lot of good stuff in there.

Greg

Views will be left in for

marcingy's picture

Views will be left in for drupal 5 and 6.

Once services is moved into core contributed modules will continue to exist.

Thanks for helping with effort to get this into core.

Agreed

bmcmurray's picture

It makes sense to me to have each Server as a separate module (whether these are eventually bundled in with Core - Optional or as Contrib we can figure out later), but that would allow a site owner to choose which servers (JSON, XML-RPC, AMFPHP, etc) they want to make available.

Something we should also address are user permissions on a per Server or perhaps per Service level; allowing certain users, user roles, or Services API key holders to access different services/servers.

B.McMurray

B.McMurray

Accessing RDF data or accessig a RDF query endpoint?

fgiasson's picture

Hi Rob,

Good job for outlining these items, however I would have a question vis-a-vis the API related to RDF.

What is the goal of the API:

1- Is it to access the RDF data (RDF data related to a blog post, a user, etc)
2- or is it related to gives information about a endpoint where people can send queries (sparql) to retrieve data?

I would think that such an API would be use to send information relative to endpoints, since there are other method to expose RDF data on the web.

Thanks for this precision,

Take care,

Fred

Both Server and Client

RobLoach's picture

Hi Fred,

This system would allow communication both ways along any medium you give it. Whether it be accessing user information via RDF, or pushing new blog posts to the server via XML-RPC. This system abstracts both server and client capabilities for any medium you want. Since this abstracts the under laying web service mediums into servers and clients, some additional functionality would be needed depending on what server you're implementing.

If you're looking for use cases of client uses for this Services API, you could use the RDF client to request user information from an external server by calling:

services_call_service('user.load', array(4), 'http://example.com', 'rdf');

You see here that we're calling the 'user.load' service with 4 as the argument. For the source, we're using example.com, and using RDF as our medium. The RDF Server module would then make the request, and return the valid user information for user #4 on example.com. Using this system, we see that it could be applied to any protocol, medium, or resource.

If we had a REST Server implementation, and wanted to make a request to the Flickr API, we could use:

services_call_service('flickr.test.echo', array('name' => 'value'), 'http://www.flickr.com/services/api/request.rest.html', 'rest');

This would effectively simulate this call on Flickr. It would call the function flickr.test.echo, with value as the parameter, calling the Flickr REST API.

Since these Server module implementations also implement the Services API for a server as well, you could make external calls from outside, and return data in any of these mediums. If you're outside the system, and want to make a request a JSON object holding information about node 6, you'd make a call to:

http://example.com/services/json?method=node.load&nid=6

And it would return a JSON object containing information about node 6. It really ends up being a powerful system and abstracts web services down to their core of being both clients and servers.

Thanks a lot, and I hope that makes more sense.

Rob

Just a thought

Richard Olsson's picture

This is just a thought I want to throw out there. Would this perhaps be a good time to evaluate the terminology used within the Services module?

The terms used by Services are easy to understand, but do not match up well with the typical (de facto, if you may) web service lingo. An example of this would be how what's typically called an operation is within the Services API called a method; The typical name for the "Services" section in /admin/build/services would be Interfaces. Server should be Binding (or at some places, Endpoint), et c.

Of course, there are pros and cons with moving away from the current terminology. As I see it, this kind of change could initially confuse old-time users of the Services module, but it could also make more sense to new users who are Web Service professionals. Of course, perhaps "Services" is easier to grasp the meaning of than "Interfaces", to someone who is just building their first WS-enabled, powered by Drupal, web application.

NOTE: By "typical", I'm referring to my own perceivings of online technical, WS-related articles, aswell as the W3C.

What do you think. Is this worth discussing?

/R

Web Services Glossary

RobLoach's picture

The W3C does cover some of these terms in the Web Services Glossary. Definitely worth more investigation.

Security

RobLoach's picture

Regarding security, Sumit K proposed a SoC project that would implement OAuth on top of the Services API. It would be good to get some reviewers on it. So go ahead, and post!

OAuth needs to be stand alone

Boris Mann's picture

As Djun has already posted feedback, OAuth is something that will of course work with Services, but it really needs to be implemented solidly and robustly on its own.

Aggregator

alex_b's picture

I haven't wrapped my mind around the services module or the proposal at hand - I just read "aggregator" on the proposal and I would like to point out that the FeedAPI project is interested in playing nice with the suggested API here and that there is a proposal for a new aggregator in D7 (http://groups.drupal.org/node/9857).

Facebook Killer or another Facebook?

csh's picture

Hear like that, D7 will be Facebook Killer or another Facebook as you are discussing.

Nothing like facebook

robertDouglass's picture

Facebook's architecture is more complicated than just a web service provider. They implement client/server-server/client architecture where both the facebook site and the application site consume and offer web services. Furthermore, the facebook site has an extensive API predicated on the parsing of their proprietary brands of HTML and JavaScript. That, with their massive caching architecture, puts them far out of reach of Drupal with a services module installed.

Informative

gdi's picture

Great post Rob. Thank You for it :)

http://www.surarticle.com

same code base?

ThePeach's picture

Correct me if I'm wrong, is this actually an abstraction layer over the drupal core, hence without modifying the MVC pattern drupal uses, or is it something more "embedded" in it?
I'm thinking about something similar, but when it comes to ontologic description of sites there're several approaches, I'm just trying to figure out how would you model it (aka just added complexity?)

A Contrarian view of REST

mikeschinkel's picture

Hi all:

I want to inject a contrarian view, contrarian at least in this crowd.

It would be really nice if the patterns you are providing where not so anti-REST. I say this assuming that most people here have probably never had their heads bashed in on rest-discuss mailing list (like I have) over their mistaken assumptions of what REST is and what it is not. After my wounds healed, I actually began to understand it, then appreciate it, and finally I become "one" with it. So much so that you'd probably now be right in calling me a RESTafarian.

For those interested in taking the effort to fully understand REST, I'd recommend RESTful Web Services from O'Reilly Media. If you want a lighter bite than that this one pager is a good place to start. And this, while on the surface appears negative, is actually in defense of REST and it explains important aspects.

With that out of the way, what's non RESTful about this Web Services API? No where in REST do you use "methods"; all things are nouns. So you would not use this as it is really just RPC over HTTP and not REST at all:

GET http://example.com/get-node/123

Instead, for REST you might instead implement:

GET http://example.com/node/123

I'm sure this seems anal and pedantic to most of you, as it did to me initially, but once have the AHA you realize it is critically important and you never want to do it any other way. What's one key benefit this provides? It provides a uniform interface making implemented code much more robust. With the URL I presented above, you can also do the following with REST where the first does an update and the second is obvious (explaining POST is a little more involved so I'll skip for now):

PUT http://example.com/node/123
DELETE http://example.com/node/123

The question is, what will it be for RPC? Chances are any two developers left to their own devices will implement using different verbs, and there's one of the many problems.

One of analogies I like to use for why the REST uniform interface is so important is this: If you were selling a toaster would you deliver it with an electric plug of your own design forcing all your customers to change their electric outlets? Of course not, that would be commercial suicide. But developers do that every day with web services. Real REST aims to minimize those compatibility problems.

So please, those of your who are managing this project, please incorporate real REST into your Web services API. Not only will you be doing yourselves a favor, you will also be gaining more respect from those influential people outside the Drupal who actually do understand and appreciate fully RESTful web services.

JMTCW.

-Mike Schinkel
http://mikeschinkel.com/blog/

P.S. Ruby on Rails went down the RPC over HTTP path at first until they finally saw the light and made things truly RESTful. We don't want to let DHH and his disciples have this leg up on us, do we?

And SOAP ?

antoine_'s picture

Hi,
This seems a really great idea : this will make Drupal web services aware and easy to extend to new protocols.
I did not see anything about SOAP. It seems that there is no real SOAP implementation on Drupal today. Are there any plans about it ?
SOAP is a W3C standard and it is widely used in the enterprise. In an intranet context, Drupal could benefit of such an implementation.

Yes there is

greg.harvey's picture

Even when you wrote this comment there was SOAP support, because I was using it to link Drupal to MS-based back office systems at Defaqto! =)

There is a SOAP client here (use nuSOAP libraries for PHP - they work better than standard PHP SOAP implementation):
http://drupal.org/project/soapclient

And there is a SOAP server here for the Services module:
http://drupal.org/project/soap_server

We found nuSOAP could deal with even the most recent SOAP standards. Didn't really use the server, so don't know how far it goes.

I'd love to look at this.

dman's picture

Recent work on taxonomy_xml has been converging on an interface that treats RDF imports and exports as a service.

I'm aiming to have a Drupal site that publishes re-usable information about its taxonomies in a way that can be imported by another site. Encourage distribution of restricted vocabularies rather than invent terminology again everywhere. Services is the way to go!

I'd love it if I didn't have to include the ARC RDF parser with my distribution and could use something close to core to build and read my data structures.

Services

Group organizers

Group categories

Group notifications

This group offers an RSS feed. Or subscribe to these personalized, sitewide feeds: