Musings on a Data API

I have been pondering the question of a data API for a while, as have a lot of people. Much of the recent discussion has focused on an Active Record approach to a data API. Now, Active Record is a very powerful architectural pattern. It maps nicely from storage to interface, it can be fairly self-documenting, and it is conceptually simple and approachable.

It is also, I believe, insufficient.

Active Record

The basic concept behind Active Record is a one to one mapping from a database record to a user-space object. That object can then be manipulated as any other object, and then saved back to the database with all SELECT, INSERT, and UPDATE statements built dynamically. Additional features, like automatic foreign key lookup or multi-load capability, are also possible, particularly in PHP 5.

Active Record, however, has a number of limitations. First is the raw one to one mapping of database fields to object fields. Very often, you want to work with data at a higher level than its database representation. That can be partially solved by adding additional utility methods to a class representing a given table or entity, but that still assumes that there is a close binding between a table and a first class entity in our data model. Take a typical Drupal node structure. While every node has a record in the node table, it frequently has multiple. Nearly every node also joins against other tables depending on the value of the type column. That's not information we can know before we load the record in the first place.

The second issue comes with multi-value fields. Multi-value fields, in SQL, are typically handled with a dependent table keyed by the primary table's primary key and either a local surrogate key or the value itself. Have a look at CCK's dependent tables for a textbook example. They are keyed by nid, vid, and a local delta value.

Classic Active Record doesn't handle multi-value fields directly well. Primarily, it is not possible to load an entire object/record with its dependent fields in one query efficiently. The primary table should return a single record, while the dependent tables could return any number of records. While it is possible to run a join on the two tables, that results in a lot of duplicate data being retrieved on each load; the primary table's fields repeat to match each entry in the dependent table. Now add in multiple multi-value fields and it becomes painfully clear that a single-query load operation for any non-trivial data structure is not possible.

That leads us to a very important conclusion: Loading a rich data object from the database will always require multiple queries per load, whether they're loaded all at once or via a lazy-loading mechanism to pull in dependent fields only when needed. Lazy loading also requires a layer of abstraction behind which the lazy loading may occur.

The third problem of Active Record is that it presumes the data lives in a local SQL database. It is, by definition, a relational-database-centric design. Data, however, is not always database-centric. Files rarely live in a relational database, and practically never in Drupal. As I write this I am working on my second project recently where the bulk of the data that I wanted exposed as a node does not live in Drupal but in a 3rd party system. In one case it was a foreign (non-Drupal) system with its own API and database connection. In the other, the data is coming in from a SOAP connection. Active Record would not have supported either scenario.

Now, that's not to say that Active Record is a bad design pattern. It is quite useful in a variety of cases, and I've written a number of implementations of it myself (to varying degrees of quality). Drupal 6 now contains the beginnings of an Active Record-style utility suite in the form of drupal_write_record(). However, for a system with the flexibility and potential of Drupal it is too limiting.

Object-Relational Mapping

If Active Record is a database-centric view of data storage, an Object-Relational Mapper, or ORM, is a user object-centric view of the world. In an ORM, the storage of an object is independent of its consumer-facing structure and interface. If it happens to map 1:1 to a database record, that is coincidental. That gives it a lot more flexibility, but also a lot more complexity.

Because an ORM is more robust, it can do a lot more optimization under the hood. Values can be moved around between tables (as CCK does) without changing its interface. Complex relationships can be modeled in a database-independent way, even tying into non-SQL systems like the file system, SOAP, etc.

The flip side, however, is that an ORM can be quite complicated for complex queries, even more so than an Active Record setup. Finding all "event nodes created by Bob in March that take place between May and July and have at least 5 attendees" is reasonably straightforward in SQL (albeit a bit verbose). In Active Record it is more difficult, but can still be done if you can access the SQL layer underneath it. In an ORM, such a query becomes very... involved. Some ORMs, like Doctrine, develop their own query language to address that problem, which in turn means parsing and interpreting a query language in user-space. PHP is not known for its string parsing and interpreting-fu. That's why SQL databases are so popular; they are.

Drupal as of version 6, however, does implement an ORM. Nodes are not built Active Record style. Hooks can add data to nodes from anywhere they want, whether it maps to a database table or not (although it frequently does). However, the ORM is "naked". That is, there's no abstraction on data access. Everything is loaded at once into a single blob-o-raw-data that gets returned from node_load(), and then it's up to modules to not bump into each other and module authors to know what to do with that raw data.

Nowhere is that more painful, in my experience, than in creating your own node form. Drupal saves the submitted form data back to the node object based on the form structure, not the node structure. Matching those up is entirely the module author's responsibility, despite the fact that the form structure and node structure frequently have no right to be the same. If they don't match up, you end up with a hybrid $form_values and $node object. That's just weird.

Thingies

Despite their additional complexity, I believe an ORM approach to be better long-term for Drupal than an Active Record approach. However, that does not preclude a hybrid approach.

What would a hybrid approach be? A hybrid approach would involve a Thingie object (that was the working name decided on in Barcelona for first-class entities like nodes, users, files, etc.) that defines multiple fields. Every Thingie is restricted to a list of fields and properties. Each field can lazy-load itself on demand. Each field is also its own object, and can either be implemented Active Record style (tightly bound to the database with lots of automation) or not (to support SOAP or XML-RPC or flat file or even calculate-on-demand fields). That is, everything becomes a self-containing CCK field.

For nodes, that would mean non-CCK fields largely cease to exist. Each field, however, would have much more flexibility and could potentially return a processed value or array. For users, the profile module as we know it gets replaced by the same structure as nodes would have, potentially even with the same field objects. Files would finally become first-class Thingies, although would be unlikely to have very complex fields. (But then, who knows?)

The ability to return complex values is important. Some modules add fields conditionally. Ideally, they should do so under a module namespaced array. That is, $node->mymodule['some_field']. With an "all CCK" model, $node->mymodule becomes an object of its own which could have its own single ->value, multiple properties, even its own methods if necessary. Some standard methods (which are more flexible than properties) would be a good idea as well for standardized access, especially for theming.

For multi-value fields, there are two options. One, all fields could be multi-value. That's how CCK works now. CCK fields have a structure of $node->field_foo[x], where x is a numeric delta. After the node has been rendered, each delta has a ['view'] property that has the rendered value of the field and then some combination of other properties for the raw data field(s). That makes handling multiple values more straightforward but can be annoying to work with when most of your fields are single-value. Two, multiple fields could be listed as separate fields. That is, if a node has a single Foo field and two Bar fields then they would be accessed as $node->foo_0, $node->bar_0, $node->bar_1 rather than as $node->bar[0] and $node->bar[1]. That would make treating each field instance separately easier, but referencing an set of fields together more difficult.

I believe CCK got this one right. The flexibility and consistency of treating all fields as multi-value is worth the more complex syntax. However, the syntax could potentially be simplified. PHP 5's Iterator interfaces could be useful here.

Objects

So far I have assumed that we are talking about a system built on classic objects (that is, objects of a class other than stdClass), not on giant arrays. Certainly, everything discussed here could be implemented using no classes at all. Any problem in programming can be solved either by procedural or OOP programming (or functional, if PHP supported that). However, certain problems are far better suited to one style or the other, where "better" could mean any of less code, more performance, easier to understand, more secure, more flexible, etc. A complex data model, such as the one Drupal now supports, is, I believe, a case where Object-Oriented principles using well-factored classes and methods is a "better" fit. As I've said before, the right tool for the right job. In this case, objects are the right tool.

That does not preclude function wrappers where a function would be more convenient. For example:

Until now there have been very good reasons to not implement heavy OO in Drupal; primarily, PHP 4's object model didn't offer any compelling reason to, as it was too primitive and limiting for the sort of functionality we would need out of it. That is no longer the case in PHP 5.2.

The other major factor is that Drupal's current architects come from procedural backgrounds, so Drupal is mostly procedural, so it attracts procedural thinkers, etc. That is not inherently bad, of course, but at the same time we should not allow that legacy style to hold Drupal back any more than we allow legacy code to hold us back.

Another advantage of a good (emphasis on good) unified Thingie system is that it would make it easier to create new Thingie types. Sometimes a one-off project could benefit from a readily-available data API that is not nodes.

Variants

In Barcelona, Jeff Eaton and I spent a fair bit of time bouncing around concepts for an OO Data API. The main idea that we struck upon was the concept of "variants". A "variant" was an alternate version of a given Thingie. For instance, Revisions of a node are variants of it. For a file, at a trivial level different resolutions of an image are different variants. However, that could easily be generalized. Picture a single File Thingie that is a YouTube-esque video. One variant of it is a high-res version. Another variant is a low-res version. Then there are small, medium, and large thumbnails of it in JPG format. Then there's an English, Spanish, and Chinese transcript, as PDFs. And so on.

For nodes, revisions and translations would be the obvious variants. Variants would also need to stack. That is, one could have node 12, revision 2 of the English version and revision 4 of the German version, but all with the same node id. That would eliminate one of the largest annoyances of dealing with content translation currently: keeping track of nodes that are translations of each other.

Each variant would have to have a default, especially since ideally we want variants to be defineable by modules rather than hard-coded. So for instance, one could load a node like:

That is, load node with id 12, the Spanish variant, the revision 2 variant, and default for all other variants. If lang were not specified, it would mean the English variant. If revision were not specified, it would mean whatever the most recent revision is.

However, that quickly runs into a problem of storage. While a nice API concept, it makes the SQL model more difficult because you then have potentially variable primary keys. That gets ugly, fast.

Perhaps those should be hard-coded...

Rendering

Although one always wants to keep data logic and display logic separated, rendering a Thingie for display is perhaps its most common use. That means we need to consider how such an API would be used for rendering. The natural knee-jerk reaction would be something like this:

It would also be very wrong. For one thing, it's very brittle. What happens when you want to render as a full node vs. teaser? As an RSS feed item? As a JSON response? Render different node types differently? Very quickly you end up with a whole bunch of flags on the render() method, and we're right back where we started.

More fundamentally, however, it is pulling rendering into the data object. That means there is poor separation between data access and the presentation layer.

Rather, we should take advantage of the fact that, in PHP 5, objects are cheap and passing objects is cheap. The data access object itself should remain pristine, but can then be passed around to other functions and objects at nearly no cost. We can also take advantage of the strict, rigid structure of the Thingie object as a collection of fields. That is, we don't need to render the Thingie! We render its fields and aggregate them for the theming layer.

Grossly over-simplified, but you get the idea. ($style would be things like "print", "html", or "rss" while $mode would be "teaser", "full", etc.)

Ready, Set, Go

One of the big drawbacks of an ORM, particularly one built on lazy-loading is the risk of the "SELECT 1+N Problem". That is, imagine the following code:

Assuming that Author and Book are two separate entity types, we will execute one query to find the books we want (with its title), and then for each book we will execute another query to load the corresponding author object with the author's name. That is a performance killer. Generally, there are two alternative approaches:

1. Greedy-Loading: The process of loading a Book or Books will also load all related data, period, with whatever joins or additional queries are needed. That means it can write more efficient queries, but it's also loading far more data than it probably needs. Do we really need to also load all of the Publisher information? Doubtful. This is actually what Drupal does currently, although it doesn't even optimize multi-object loads (unless you're using a List View, which then acts as a very optimized but complex ActiveRecord-ish loader, not an ORM).
2. Semi-lazy loading: If you know in advance that you're going to want the Author object as well, you can specify that the ORM loader should also pull that data. That is, you write a JOIN, but you don't use SQL but some ORM-specific directive, usually a very verbose one. As I see it, that loses much (though admittedly not all) of the advantages of a lazy-loading ORM: Don't make me think. Then if you need to pre-load against multiple other fields, well, pretty soon you have code so complex that you may as well just write your own SQL and be done with it.

And neither handle multi-value fields, of course.

I would propose a third, more flexible approach; let's take a cue from both jQuery and CCK. A jQuery object (the $() thing) is never a single entity. It is always a set of DOM nodes, sometimes a set of one DOM nodes. In CCK, a single-value field is, structurally, the same as a multi-value field that only happens to have one value. Views are all about multi-object queries, although those can also frequently have only one entry. They're still treated as multiple entries.

Set operations are both extremely powerful and very natural. SQL itself is built on set operations. Many RPC requests either assume or allow set operations. There is, in fact, no reason why a single-entity operation cannot be implemented as a set operation.

OK, not entirely true. If you have a naked data structure, then you never have an entity but an array of entities, which you then need to foreach(); that can be clumsy if you don't practically use multi-value fields often. That's another place where having an OO interface is "better" (in this case easier) than a procedural one. You can encapsulate the loop, much the way jQuery plugins do. That makes the module-developer-facing API cleaner. This is a good use-case for Interfaces:

But why do we even need a loop? Moving a loop inside a function call makes the API cleaner, but especially when dealing with SQL databases doesn't necessarily help performance. Remember, though, that we're demanding a rigid Thingie structure. A Thingie is composed of Fields. Let those Fields handle their own set operations, especially for lazy loading. That means each field needs to know what its "load set" is. That is, suppose we have a Set of Node objects. When we access, say, the taxonomy field on one node, it should lazy-load all taxonomy data for the nodes in that Set in a single query and then assign out the data to the appropriate node's field. Update operations then can also be grouped, making it possible to load a set of Thingies, set one field on all of them, and then save, with the load and update operations both taking only one query each no matter how many Thingies we're modifying, and without writing any SQL ourselves.

I've implemented such a system recently for a 3rd-party integration project, and it is actually not difficult to implement. (Well, the read part, anyway. The write part is more complicated, but I believe it is solvable.)

While not as efficient as explicitly specifying the fields we want, it does flatten the cost. That is, the Book and Author example above would have a cost of 2: one to mass-load books, and one to mass-load authors. The worst-case scenario for such an approach is that every possible field gets called anyway, or 1 + F. That is exactly the cost of a node_load() in Drupal now... except that node_load() is actually N*(1+F) because it doesn't do set operations.

Searching

The other major drawback of an ORM is that non-trivial searching is frequently very verbose. We need to specify a variety of weird search parameters in a completely data-agnostic fashion. That means lots of methods, or lots of objects passed into a long method signature with lots of arguments, such as in Qcodo. Or it means a custom query language like Doctrine. Both are icky.

To be honest, I see the best course of action here to be "punt". Separate loading from searching entirely. A set-load operation should take simply an array of IDs. How those IDs are found is a separate question. They could be from a simple API call, or looked up from a field in another object, or from a Views-like builder, or from a direct SQL query builder, or a hand-rolled 15-line super-complex SQL query (which would be about 1/10th the size of trying to write that query abstractly with fully-OO directives), or retrieved from a remote server via SOAP. By keeping searching and loading separate, we maximize flexibility while minimizing complexity. While there will be, in some cases, a cost in terms of additional queries, that cost is relatively flat and, I would argue, a good value.

Now, combine flexible searching with everything-as-a-set, and you have a very large portion of Views not just in core, but being core itself.

What not to do

There are, of course, a lot of potential pitfalls in any such endeavor. There are two I will mention in particular.

1. Deep inheritance. Deep inheritance hierarchies are well-understood as being dangerously brittle. Adding functionality requires editing existing code. You can't do things "sideways", as Drupal does nearly everything. If you want a certain piece of functionality to be part of two classes that otherwise have no reason to be logically children of each other, you're SOL. That's not to say that inheritance is always bad, but it is very easy to get carried away. Let's avoid that.

   For example, it may seem natural to have a Node class, and then child classes of Node for Page, Story, Event, etc. However, that's not how nodes are actually used in Drupal. A node is a node is a node, but it has some combination of fields on it (even now). Rather, a Node should continue to have a ->type, which in turn indicates what field objects should be added to a generalized field container.

2. Code generation. I have never understood how the "Don't Repeat Yourself" and code generation design philosophies can coexist. Code generation (speaking of logic generation here, not function skeleton generation) is repeating yourself. In fact, its repeating yourself so much that you make the computer repeat yourself for you. That's a sure sign that you are doing something wrong, and need to refactor your code to actually use inheritance, or composition, or something other than typing out the same code over and over again. Of course, there is one catch here: PHP 5.2 and earlier does not support late static binding, which means, in essence, static methods can't be usefully inherited because references to other static methods will always be to the defining class, not the actual class. Fixing that is a hotly debated topic for PHP 5.3, but that's still well-off as far as we're concerned.
3. Tight-coupling. (Nobody expects the Spanish Inquisition!) A truly pristine data API will require fully and totally divorcing it from the rest of the system, particularly forms. The node edit form's save routines, for instance, should not in any way shape or form know anything about SQL. They should all simply set values on the $node object, and then eventually $node->save() gets called and fields internally save themselves. Similar separation is also necessary for theming. (Like, can we get comments out of node_view(), please?)

The future

What I have described here is, of course, ambitious. Very ambitious. It represents a fundamental shift in the way Drupal handles data. However, I believe that shift is necessary for Drupal's long-term success.

Dries has set some rather lofty goals recently. Web services. Drupal as the "Linux of the web". He doesn't think small, and neither should we. A robust, powerful, and above-all decoupled data API moves Drupal from being a CMS or application framework to being something else: A data server. The menu system acts as a simple router to handle RPC/REST requests, which the data server responds to. It just so happens that the most common REST request is for an HTML page; at least for now. In 5-10 years, though? Will we even be talking about SQL and HTML? A good data API should be agnostic of both.

Implementation

So what would such a system look like in practice? How would we use hooks with it? I'm glad you asked! I'm not entirely sure yet, aside from the samples above. I have some ideas, however, most of which owe thanks to other members of the Drupal community. For instance, courtesy of Jeff Eaton (who may credit it elsewhere himself) is hook_nodeapi() going away in favor of hook_node_info() defining what field types a given node type has, and then those field types (objects) internally do whatever loading and saving they need. Adding fields to a node type is accomplished by a simple hook_node_info_alter(). hook_user() would likely suffer a similar fate.

One of the possibilities discussed in Barcelona was to implement just files using a new Thingie-based system, as a trial balloon. Despite recent improvements, Drupal's file handling is still a weak point. That means the risks of experimentation are lower than if we were to break, say, nodes. If it works out, we can then extend it to the rest of the system in the next future, and we'll have a good API behind which to implement things like, say, pluggable storage engines.

At this point, I welcome feedback from all and sundry on the concepts presented here. Truthfully, I do not expect it to all happen in Drupal 7. If it all happens by Drupal 8, it will be aggressive. However, we do need to have a clear picture of where we are going and how to get there, even if it will take a few versions.