Proposed Web Service Clients module: Preparing the way for external fields

At the recent Data Architecture Design Sprint, participants outlined a development path in which fields can be local or external. A given entity instance (node, user, etc.) can be fully local, fully external, or a combination of the two. For example, a node can have several fields stored and handled in a local SQL database and other fields both loaded from and written to an external SOAP service.

As steps on this path, we outlined (a) short term changes to CCK, (b) a minimal patch to integrate CCK into core, and (c) some follow-up steps, like deciding on a consistent data structure for all entity types, to be followed by a unified set of methods for all entity types (e.g, drupal_save($entity_type, $entity)).

Are there short term steps we can (should?) take now toward external data source support? Here is a rough, conceptual attempt to sketch out such steps in the form of a proposed contrib module, Web Service Clients, taking the existing Services module as a model to work from.

Services module: a model for external data source support

In D5 contrib, a services module was developed to enable standardized development of Drupal as a web services platform. Documentation: http://drupal.org/node/109782, project: http://drupal.org/project/services. Services implements a pluggable architecture in which various types of servers (XMLRPC, SOAP) can be implemented, drawing on a common set of services. Services here are sets of available methods, e.g., node load and user save. Through services, a Drupal site can respond to external requests, providing read/write access to locally stored data.

Clients module: web service clients modelled on Services

Services provides external read/write access to locally stored data. We need the parallel set of implementations: local read/write access to external data sources (via web services).

Parallel to the Services module's pluggable server, we need a set of pluggable server types--but as clients rather than servers. In short, we need a Web Service Clients module (Clients for short).

A minimal initial target spec for this module might be:

• Using XMLRPC (the one server plugin bundled with Services module; there are others in contrib), implement a client such that a Drupal site running Clients can meet the following use cases:
  1. User can create and update a node that has two fields from an external site running Services' XMLRPC server. On create and update, those fields are not saved locally but instead are saved to the remote service. On load, fields are loaded from the remote service.
  2. Whenever a user record is updated locally, a call is sent to a remote service notifying of the update operation. This enables a remote service to store/track a user's ID and associated email address on the Drupal instance.

Solution components:

• Generic handlers (in clients.module): shared methods
• Pluggable services: services supported, e.g., XMLRPC
• Pluggable methods: local methods for handling data operations
• Client instances: data unique to specific client instances

Generic handlers

Generic handlers and methods, including:

• Connecting to external services.
• UI for admin-defined client instances.

Pluggable services

Example: generic SOAP support.

Each service includes methods for:

• authentication
• deriving schema information about external data sources

Pluggable methods

Define in metadata form a common set of local methods and their expected data formats and return values. Examples:

• entity_find: given an array of properties and values, return an array of matching entity IDs.
• entity_load: given an entity id, return an entity instance
• entity_save: given an entity instance, save to the remote service and return a result code
• field_load: given an entity id and a field name, return the value of that field for the given entity
• field_save: given an entity id, field name, and field value, save the field to that entity and return a result code

Alternately, we could implement the same set of methods as are available in Services module (see the services plugins), but these may tend to be too specific to the Drupal environment and so to map poorly to external web services.

Client instances

Example: SOAP client to a particular SOAP service.

A client instance may override generic methods for its service type.

Each client instance includes handling and methods for:

• Mapping available external request methods to the set of local methods. For example, fed a field_load call, translate this into a call to save data to a remote SOAP service. (Each client instance will implement only a subset of the available local methods.)

Nodeapi and CCK integration

To locally present and edit data from external sources, we need the ability to represent and integrate those data locally.

CCK provides our richest set of applicable tools, and is in any case our proposed long term solution. How therefore can we leverage CCK fields in 5.x or 6.x to provide handling for external fields?

Proposed implementation:

One-to-one mapping between external data fields and local CCK fields, with the CCK field handling overridden for CRUD operations.

• For each external field, site admin defines a CCK field and attaches it to one or more content types.
• Clients module provides additional UI element in field definition to select external field to map this field to.
• In hook_nodeapi() load op, Clients module overrides CCK field handling, inserting externally-derived data in place of local ones (which will be empty, as there are no locally stored data).
• In hook_nodeapi() insert, update, and delete ops, Clients module dispatches calls to external web service for fields it is responsible for.

Actions

As well as field-level handling, we need methods to invoke actions based on given state changes. In the above use case #2, this included responding to a user save operation by sending a notification to an external service.

Here we can build off of Actions, particularly the hook operation support in D6.

• Client method implementations (particular local methods, e.g., entity_save, that are implemented by a specific client, e.g., a Gazetteer SOAP service) may declare the 'hooks' operations they support, using the same format as used for actions metadata.
• These methods are programmatically exposed as actions with associated hooks, to be called as appropriate, receiving in argument form the relevant data structure (e.g., a user object).

Summary

By developing genericized, fields-aware web service client support, we can both provide short term functionality for D5 and D6 and begin concretely to map out long term solutions for core.

After getting our minimal CCK fields into core, we could turn to minimally bringing the Services module and (assuming some solid development by then) accompanying Clients module into core as well.

Does this approach seem to make sense? What considerations are missing? Please comment on or edit up this Wiki page.

[Note: I'm hoping to be able to begin coding a draft Clients module within a week or two for an upcoming client contract.--Nedjo]