Data APIs: building a spec, defining options

[Here is an attempt to pull together some of the information and discussions on the go in a way that might help focus our discussion. Please edit/add/expand.]

Assume we're starting from scratch and writing a set of APIs for data handling for our main object types (node, user, etc.). What does such a set of APIs need to do?

Currently we have a diverse set of data APIs. A useful first step in designing a new unified set of APIs may be to summarize what we currently have. From there, we can identify key gaps and compare options.

Characteristics of existing data APIs

What are the characteristics of (one or another of) our current main object types (nodes, users, etc.)?

Under our current data APIs, an object:

1. May exist in numerous different instances
   Example: node revisions.
2. May have subtypes, each with its own specific set of properties and behaviours
   Example: node types
3. May represent data stored in various ways
   Example: node attributes stored in SQL database
   Example: files stored in the file system
4. May have many to one and one to many relationships with other objects
   Example: multiple nodes authored by a single user.
5. May have many to many relationships with other objects
   Example: a node has many terms, a term has many nodes.
6. May have a set of built-in properties
   Example: node created time.
7. May be extended with sets of defined properties
   Example: user 'categories'
   Example: event properties as implemented in event module
8. May be extended with defined properties attached to sub-types
   Example: CCK fields (limited at present to object subtypes--may not be attached directly to object types)
9. May be extended with arbitrary properties at run time
   Example: user object's 'data' field.
10. Are not limited to Drupal core but may be created by contributed modules
    Example: Views
11. Provide structured intervention points, enabling other code components to respond to data events
    Example: user object's hook_user().

Questions on existing data APIs characteristics

1. Do we need to retain all of these characteristics?
   If we look for a moment at our user APIs, they are shaped to a considerable extent by the aims of (a) being extensible through 'categories' of properties and (b) accepting run-time properties of any sort (as stored in the serialized 'data' field). Are both of these characteristics necessary?

   1. very useful for everything. currently hook_user _node and _comments make this possible in different ways. - chx.
   2. can be dropped IMO - chx.

2. Which of these characteristics should be available to all object types, and which (if retained at all)should be limited to specific object types?
   E.g., should every object type accept sub-types (like nodes currently do)?

Where does the thumbnail of a file fall? It belongs to one specific object (so it's not like node types where nodes under different types are autonomous) and it's active (unlike node revisions which are normally not visible).

Gaps: additional characteristics and features

In addition to currently-met criteria, the following have been identified as desirable object characteristics/features.

Data APIs:
1. Are not limited to a single type of data handler (e.g., SQL database storage) but may be handled in multiple ways
E.g., a node can be loaded from and subsequently written to an external data store (in XML format).
1. Are uniform and consistent across object types
A single set of methods or common object methods may be applied to different types of objects.
1. Are schema-aware
Rather than consisting of e.g. a piecemeal set of hard-coded SQL statements, object handling is introspective and derived from a structured representation of the object and its relationships.
1. Are fully autonomous of user input
Data handling is fully abstracted from any particular input process (e.g., forms).

Existing model implementations

Within Drupal development, several initiatives have sketched in approaches potentially relevant to our work. See the list under "Code" at http://groups.drupal.org/node/8235 (the first two are less relevant).

What are the relative merits of these draft approaches?

CCK fields

Of the various existing code pieces, CCK fields deserve special attention. They are the most advanced, fully developed, and broadly used. There is a clear need and demand already to bring them into Drupal core.

Can CCK fields serve as a model for a unified set of data handling APIs? Can/should all object properties be handled as fields?

What changes would be needed to CCK fields to meet the full set of data API specs identified above?

1. Further abstract from user input
2. Ability to apply fields directly to objects (e.g. replace the existing profile module functionality)
3. Handling of fields in the object's primary table, e.g., node status, user created.