Data Architecture Design Sprint

First fields in core issue + core

bjaspan — Mon, 02 Jun 2008 14:21:50 +0000

Please see http://drupal.org/node/265604.

Field API, field structure, and data migration

bjaspan — Wed, 14 May 2008 22:08:07 +0000

Much has been said (http://groups.drupal.org/node/9297) about how fields should be structured in D7 core, what aspects of fields can be changed, and how those changes are implemented. It is (past) time to move forward on implementing fields in core and in this post I am proposing an answer.

Disclaimer: I make a lot of declarative statements in this post. Obviously I do not have unilateral authority; this is just a proposal.

Field storage

Contrary to what we "decided" at DADS, D7 will continue to support per-content-type and per-field storage as we already do. I spelled out my reasons at http://groups.drupal.org/node/9297#comment-37050. The short summary is "performance"; until someone measures otherwise, we're assuming that fewer joins are more efficient than more joins. Since we already have the code to support both storage methods, the implementation cost is low.

The Fields API

At DADS, we declared/clarified some basic concepts:

A Field Type is a data type implemented by a module: text, nodereference, address, etc. The module defines semantics and functionality.
A Field is a specific configuration of a type. Fields have settings that define the unique characteristics of that field. All fields share some properties, such as: field type, name, cardinality (currently "multiple"), sharable. A Field Type can define addititional properties; for example, a the Text field type defines the property "formatted" which can be FALSE (plain text) or TRUE (user chooses input format).
A Field Instance is the binding of a Field to a Content Type. Field Instances have Settings that related to the association, such as: display name (label), weight, input widget, display format.

A field instance's settings do not affect the field's underlying data and can be changed without altering the field. By contrast, a Field's settings cannot be changed because changing any of them constitutes fundamentally altering the field's identity. However, a field can be "migrated" to a new field (of the same or different field type) to provide new functionality or semantics for the existing data. More on this later.

D6 CCK created the first version of the Fields API. It is a big step forward but does not cleanly separate the concept of Field and Field Instance. We currently have the function:

<?php

function content_field_instance_create($field); ?>

The $field argument contains information defining both the Field and the Field Instance. (Almost) Any property of a field or setting can be specified. create() will both create the Field if it does not exist and then the Field Instance; if some Field Instance settings are not provided, they will be inherited from an existing Field Instance if one exists, the Field settings, and system-wide defaults. If a non-shared Field Instance of the Field already exists, the Field is immediately converted from a non-shared field to a shared field, requiring a change in database schema. And so forth. content_field_instance_update() is similar. A lot happens inside these functions and I think their complexity is is keeping us in a straitjacket.

In D7, the Field API will cleanly separate Fields from Field Instances. We will have two create functions:

<?php
function content_field_create_field($field);
function content_field_create_instance($instance);
?>

create_field()'s $field will contain field name, field type, and can accept all global (e.g. field type, name, required, cardinality, sharable) and type-specifc (e.g. formatted) field settings. create_field() will probably do nothing but store the information in the content_field table.

create_instance()'s $instance will contain the field name, the content type to bind to, and all global (e.g. label, weight) and type-specific (???) per-instance settings. create_instance() cannot change any Field settings (e.g. sharable, cardinality); those settings are simply meaningless in the $instance argument. create_instance() also imposes the Field constraints; for example, if you try to create a second instance of a field created as non-sharable, create_instance() will fail.

The Fields API will also have some fairly straightforward functions such as:

<?php
function content_field_rename_field($old_name, $new_name);
function content_field_update_instance($instance);
function content_field_delete_instance($instance);
function content_field_delete_field($field);
?>

Note content_field_update_instance() will be "straightforward" even though the current implementation is not because like create_instance() it enforces the constraints of the field; everything it does is pretty light-weight. The heavy lifting is reserved for "field migration."

Field migration

Conspicuously absent from the functions above is:

<?php
function content_field_update_field($field)
?>

This is the magical function that can convert fields from shareable vs. not, from cardinality 1 vs. N vs. unlimited, from "plain text" to "formatted text", and from "text" to "nodereference". Notice that these are all changes of Field settings, not Instance settings.

As I said at http://groups.drupal.org/node/9297#comment-37050, Drupal needs this functionality because humans cannot predict the future. However, Drupal core does not need this functionality. Fields in core provide Drupal with capabilties not previously possible, but changes to fields in core is fundamentally a development-time operation that can perfectly well depend on contrib, as it does now.

update_field() will handle three kinds of updates:

Changes to shareable and cardinality. This is the code that moves columns between per-content-type and per-field storage and adds/removes the 'delta' column. It is the (only) code that would be unnecessary if we declared all field tables to use per-field storage, and it is code that is already written.
Intra-field-type changes such as plain-text to formatted-text.
Inter-field-type changes such as "text" to "nodereference" (a.k.a "the DabbleDB magic").

It is important to recognize that #2 and #3 are actually the same thing and, despite what many think, CCK CANNOT CURRENTLY PERFORM EITHER ONE. Yes, the one special case of plain-text to formatted-text works because it is a degenerate case of "add or remove a single column, changing nothing else," and maybe there are some other similar cases. But you can't convert an ISO Date field to a Unix Timestamp Date field because the CCK doesn't know how to do that with a simple assignment or typecast without any knowledge of the underlying field types.

The core content_field_update_field() function will work by dispatching these operations to modules via hooks. I am not yet 100% sure of the interface but it will look something like this:

Shareable and cardinality are "core" field properties and can be changed in a field-agnostic manner; the actual column types, names, and content always remain the same (though some content may be discarded), they just move from one table to another, possibly with/out a delta column. This will be handled by a dedicated hook that is implemented by the CCK UI module. An admin performing this operation will have to have the CCK UI module installed anyway; a module developer wanting to perform this via an update function will just have to depend on the CCK UI module (they do now anyway).

If you try to update a shared field to be non-shareable, it will fail; explicitly delete all but one field instance first. If you try to reduce a field's cardinality, I'm not sure if it should fail or simply silently discard data (which is what it does now, I think).

I can easily imagine this functionality, which is mostly already written, being moved into the core Fields API at some point when we resolve the PHP timeout and race condition issues. If we ever want to change a core field's shareability or cardinality, of course, we'll need the core in core then (as an example, this would be the case if we decided that nodes can only have one term and it lives in the node table). For now, we do not need it in core.
Changes to field-type specific properties (e.g. "formatted") or field-type changes will be implemented via a hook like:

<?php
  hook_content_field_update_field($old_field, $new_field);
?>

The first module to return TRUE says the update is done; if all return FALSE, the update is not supported. Fore example:

If $old and $new are both of type 'text', presumably text.module will accept the update: change the formatted property by adding or removing the format column.
If $old is an ISO Date field and $new is a Unix Timestamp Date field, date.module will accept the update: create a new field, execute "UPDATE new_field.value = TO_UNIXTIME(old_field.value)", delete the old field, and rename the new one into place.
If $old is any type and $new is type 'nodereference', presumably nodereference.module will accept the update: create a new node type, SELECT DISTINCT on the old field, create a new node for each value, call content_field_create_field() to add the new field, call content_field_create_instance() for each appropriate content type, UPDATE the _nid column of the new field based on a join of the old field columns to the node table for the new node type, call content_field_delete_field() on the old field, and cannot content_field_rename_field() to move the new field into place.

Or something like that. :-) CCK can't do this today so I do not feel too bad about not having ironed out all the details yet.

Data Architecture Design Sprint

Remote Field Revision

neoliminal — Wed, 05 Mar 2008 17:14:56 +0000

This discussion will revolve around issues of Remote Field Revisions. This regards getting field population from single, multiple, and unknown sources. Sending data to other sites and how they can use, modify, and potentially change that data from source. Also regarding race state and origination of data sources and trusted sources for field revision.

Data Architecture Design Sprint

Drupalcon: The Future of Fields

kentbye — Tue, 04 Mar 2008 18:32:40 +0000

Here are my notes from the Future of Fields presentation from this morning.

[Begins showing a picture of the Data Architecture design Sprint including: chx, yched, bjaspan, crell, karens and nedjo]

Drupalcon used to be a developer conference where they would meet and figure out what was going to happen for the next year, but in Barcelona they realized that it was harder to do that.

So they asked about the who are the right people to be involved, and they all flew into Chicago and did a sprint.

Barry would advocate this be a trend in future Drupal development, get together a targeted group to make some progress.

The DADS (Data Architecture Design Sprint
Re-design of Drupal's core date architecture
* Data API
* Object modeling - node object is a dumping ground for stuff. Some is in arrays, some are in classes, some are fake classes.
* Fields in core -- What does that mean? CCK in core. Once the node object is cleaned up, let's clean up how it's rendered. Then realized that started getting really big, and started to focus on getting fields into core, and then things would go from there.
* Many related ideas

Their goal was to propose a design at Drupalcon

Our "Grand Conclusions"

Some key thoughts:
What makes Drupal unique?
Drupal has the hook-based system
Drupal's architecture allows contributed modules to easily add value to content
Everything from fivestar to views -- and that there are so many of these modules, and that the fact there are so many of these modules is what makes Drupal unique.

The Future
Web services -- sharing data from multiple sources
* Consuming - Need to import in Amazon data or RDF
* Providing access to our data -- no way at the moment to export data via JSON without custom coding.
* Enhancing -- Being able to do any Drupal action on fields that is currently limited to nodes

Drupal is all about our contributed modules, we need to be able to provide our secret sauce throughout Drupal. Otherwise we're just sucking and aggregating other people's data, and not leveraging Drupal's power and flexibility.

So we want to add Drupal's value to the data sent out to web services. If we take photo from a Flickr, it shouldn't have to know it's a node to be able to vote on it. And putting CCK into core, that's a big step towards that.

[Showing KarenS flowchart]
Drupal takes in data, formats and saves data, and produces HTML (and XML, etc) -- The little triangle at the bottom are the Drupal hooks that holds Drupal together.

If getting fields into Core is where we want to go, then Karen will talk more about it.

KAREN STEVENSON: We all know that getting fields into core is the big issue that we want to do for D7.
So why bother? The way CCK handles fields is what we want to do, and it's a good way of looking at the issue.

At the usability testing at UMN, people were confused that Title and Body were treated as fields -- even though they don't act the same way as fields. The reason why they behave differently is that they're not in core. One way to consistency is to get fields into core in order to have consistency.

In a way, we have developed something better for users than developers. Users can create fields on the fly, but developers need to be able to do that as well. So that's also a motivation to get CCK fields into core.

Hopefully people are familiar with CCK.
"Field type" is how is it stored, and what is unique about it. A field type tells you how it is stored in the database.
The "field" is really the settings, how we set up the field, the length or what type of field.
"Field instance" is how bind a field to a content type. The instance also have settings.

It's confusing terminology, but that's the way CCK works now.

Let's take a look at existing things in core to see what types of things could be fields.
Body: text area with a teaser splitter. It's a two-field, teaser area and body area.
Created: date field
Upload: file field
User picture: image field
IDs: Number + optionwidgets
Taxonomy feels like a field -- but taxonomy has so much special processing, and will be a challenging thing
Comments? Fields? Or are they data? They're not sure yet.

Several of these things are not even in core CCK like date, file or image field. So it's a double promotion from not existing in contrib to not existing in core. And the reason why they're not in CCK is because they're not simple fields, and all of those things should make it in core CCK.

When we do a split between -- no one is talking about taking ALL of CCK and putting into core. That's too much, and would be too much complication

What is the Minimum Viable Possibility of how much should of CCK should go into core D7 and what should stay in contrib:

Core
* Field API
* Field Storage Engine
* Node CRUD
* WIdgets
* Forms
* Formatters
* Multiple values -- handle them as a whole or as a separate things: like a GMAP
-- "Add more" -- Core now has add more button
-- Custom
* FIeld validation

Contrib
* Field UI -- that'll have to stay in contrib, because it's messy and hard
-- Add, manage, diplay tabs
* Fieldgroups
* Allowed & default values -- probably more complicated than we need to go
* Content Copy
* Module integration
-- Views, token, pathauto

The core module would be called "field.module" and we continue to have a contrib module.

Field storage options
CCK does dynamic storage of fields, it is done in a per content type, and a multiple content is put into a 'per field'
If you share a field between two content types, CCK has to create a new field.

Any time you change parameters, you have to alter the schema, and move the data from one table to another one, and potentially loose some data

Lots of places where it can go wrong and where data could get lost, and it's worse now because of CCK has a dynamically-defined schema. Store the field information in a field table, and whenever there is a call to determine the schema
Ran into race conditions -- if you want to actually change the schema.
And the big concern is to get all of this into core without breaking everything and making it a maintaining nightmare.

So fixed 'per tables' tables with cached node array
* Simple structure and code
* Downside is that there is a performance hit from numerous sql JOINS

One solution is to combine 'per field' tables and intermediate tables better optimized for query
* Duplication of data, yields a larger database

They are doing a separate cache for the node object,so that they're not doing the joining as often -- which works fine for node_load, but we're back to lots of JOINS with views. And currently Views is already doing the JOINS, but they've been going around and round -- it's been a show stopper without figuring this out.

Their preliminary conclusion os that simple fields seems to make sense. It gives simple code that's predictable, and let's find out if these JOINS are an issue with performance tests. They imagine coding it up and implementing it, and then seeing if there are killer performance issues. If so, then scrap it and try another approach.

The Field API
There is a basic API in CCK D6.
Having the ability to have modules to create fields creates some problems.
* Should the fields be locked down and not be able to be altered?
* Who is going to own the fields?
* Are they available in the UI?
* Should those fields be shared?

A lot of these issues and more need to be worked through

In CCK for D6, they are working on how to get some field-based permissions, and so it may also be a Phase II issue for getting these permissions fields into D7 after a big chunk of functionality has been committed in the Phase I.

If they had permissions for fields, then modules could do more things with fields. Jeff Eaton has been in discussion with KarenS, and they've agreed that it's better to start with them as being locked down permissions-wise, and then gradually move towards allowing people to have permissions on them.

Possible Roadmap

Can start the process as separating the API from the UI and create a new fields.module in D6 with the idea to move it into D7.
It'll have to include the basic fields (text, number, optionwidgets)
Rework existing core fields and transition them into fields.module -- they think it makes sense to start with the Body field.
Determine what else needs to be done for date , image, file handling etc.
If they can get something into D7 early in the process, and then possibly get some of the UI into core.

QUESTION: Would Page and Story define in the CCK way instead of the way they are?

Possibly.

The revisions table could go away because the body would become the revisions table (and CCK currently handles revisions on the field level)

QUESTION: If fields get into core, then instead of Webform would we now just use the core field.module?
Not sure, a lot of things could happen.

QUESTION: What about the Image field -- Possibly grouping fields for the alt tags, etc. to have the same table? -- And a fourth option, not dynamically from the UI, but that there is a programatic way to have a hook that says that these fields will always stay together. You could keep consistency in the schema API

The API will be able to define how fields will be grouped together.

LARRY GARFIELD gets up to talk about Local Fields on Remote Content

One challenge: If you want Drupal to talk to foreign data, then it's really, really hard.

What do we mean for data sources?

local nodes
Entire Amazon catalog
Flickr photos
legacy data -- accessible via SOAP, XML, etc.
Other content from Drupal sites

For any given piece of content you want to perform the any of the following operations:
* Display locally
* Comment
* Vote
* Search
* Views

These operations don't work for non-local data.

Importing ALL of that data can be really bad, because you have huge amounts of data -- and even if it was possible, then keeping it in sync could be really crazy. Lazy node creation also isn't much better.

But with everything we've talked about so far -- we don't get anything new and exciting. So what would this enable that we could really get excited about?

The Data Architecture Design Spring participants talked about "Institute of Contempory Art" as a case study -- coincidentally is very similar project to what Palatir was working on.
* Data in legacy database that was available via SOAP, and that they didn't want to import b/c needed access via another non-Drupal CMS.
* They had artwork (title, year, image ULR) artist (name birthplace, bio), resources (video, audio, etc)
* Had to figure out how to make all of this available to Drupal.
* The goal is to have the http://example.com/artwork/abc -- and it's not a node in the database, but they want to be able to treat it as a node.

So what does it mean to be a node?
It has a nid, and some simple properties like created, status, and simple metadata, and then they have the Fields, which are the meat of the content -- title body, comments, taxonomy, CCK fields, and other complex structured data attached on by other modules.

So how can you provide a unified structure for the Node and the ICA: "Thingy"? [Thingy is a temporary name decided in Barelona that won't be called a thingy when it gets into core.]

It has a unique ID: that has an opaque string that has properties (i.e. metadata), and it has fields -- which could be intrinsic to the title or body, as well as extrinsic fields that come from the Drupal database.
Anything else that is used that is pulled in doesn't need to be known where it's coming from remotely -- loading the object we shouldn't care where the external data is coming from. They want to have a clear separation between the data source and the data interface.

How do you add a field to a thingy so that you can add Fivestar.module and add some votes? Instead of a single column for a nid, you have two columns, you have the the type of nid (node, artwork, artwork), id (12, abc [menu path on the file system], abc), Delta column (0,0,1) and then the Vote (2,5,4)

The artwork "abc" does not exist anywhere exist anywhere else in SQL other than this vote column.

The Delta column is a CCK-specific value that allows for multiple values for CCK fields.

QUESTION: What if you have a combination of some images from Flickr and some locally
Having a given field with having some values come from some data sources would be helpful, but would be really difficult to implement. The heavy lifting is on the field level.

QUESTION: What if the remote data is deleted, how do you sync?
We don't know yet, that's yet to be determined, and part of the problem with dealing with external and stale data.

The thingy called "artwork" is equivalent to what we currently call a "node." Artwork would be a class, and they we load the object with "abc" to pull it in from the remote system, and the only thing stored locally is the Drupal value-data like a fivestar vote.

QUESTION: Works well from files or images, but what about text strings and caching?

It would have to be exposed in some form that you can query. But depends on your use case.

This group left some of this discussion out of the presentation, and have more info in their group on g.d.o.

What about searching these?

Can't search local database AND remote SOAP within one SQL request -- possibly with SPARQL, but will probably still have to do a separate query for each data source and merge results.

Views -- probably does not work as we understand it today because Views is tied to a local SQL database. We haven't figured it out yet -- possibly lazy load SQL. Will probably have to just implement it and see what works and what doesn't work.

The Views query-builder will live on for local data, but the rest of views is unclear for how it will evolve or adapt for dealing with remote data.

NEDJO ROGERS: Brief summary of these ideas and proposals, and we need to hear back and have follow-up discussions for how much this makes sense. They will have to fundamentally rework how core works with data with nodes and users.

We also have some assumptions in Drupal that are being challenged:
* All data is entered into the dB via forms for users
* Also anything can be extended at any time.
* All data resides in a local SQL database

CCK knows what it's object looks like, we don't have to guess b/c it's aware of it's schema. It's sort of representing the type of data API we're looking for. Let's do it, but let's do it right. By renewing our core Data API so that we can free our Data API from these assumptions.

Some of these things can be taken care of now with CCK for Drupal 6.
Looking at a minimal implementation of CCK for Drupal 7, and where we take it is an open question.

They'd like to hear back any questions and comments, does it make sense to people?

Does it ring true as a future direction for Drupal, and use the CCK fields as a broader goal for some of these things moving forward?

None of this is going to happen unless someone takes ownership that is going to get involved and help make it happen.

If you want to help, and don't know how, then write unit tests so that it will extend the development cycle.

Much more info on Daily reports, Final Report, Proposed Content Models 1 and 2, and more ramblings:
http://groups.drupal.org/data-architecture-design-sprint

Replace the concept of nodes with objects that can be uniquely identified -- how we do it: whether it's something on top of the node vs. something else, then take a look at Model 1 and 2.

QUESTION: Will the data structure that is abstract and flexible enough to have a choice of fully normalized vs. de-normalized with no joins.

We don't know yet. So when we think about how can we attach a comment to an external Flickr photo, there is a choice between whether the field table knows what the actual data looks like or another option is something actively being discussed.

They do have adding fields to fields in D6.

QUESTION: Stale data problem, and data unavailable.

Might be implementation specific: dependent on the field on whether it's a Flickr photo or for the museum. But what you say generally is "data unavailable"

QUESTION: How would node revisions work if KarenS is saying that using the field.module for the body would get rid of node revisions. Do revisions go away? What about the diff.module?

Every CCK field keeps it's own revisions, and so she's is throwing that out, because CCK has something very similar already built in for each field.

QUESTION: Also think about the possibility of storing the revisions as a diff, so that you have a whole version control system built in internally.

QUESTION: Why not just use XML, and then think as thingies as 'entities' and fields are 'attributes'.

It's good for some tasks, but it's not really a good mechanism to using it internally. XML aren't multi-value. You could easily map the thingy structure to XML if you want to export.

Data Architecture Design Sprint

Field Structure

KarenS@drupal.org — Sat, 01 Mar 2008 12:18:23 +0000

The way that fields are structured in CCK now is that any field that is multiple or shared has its own separate 'per field' table, and all other fields are grouped together in a 'per content type' table. Querying this data to create a node can be expensive, so the serialized node is cached and the cached data is used during node_load().

That structure requires a lot of complex data manipulation when fields are shared or unshared or changed from single to multiple or multiple to single. When those things happen, the schema must be altered and data must be migrated from one table to another. This makes the code quite complex and introduces the potential for data loss and other errors.

Our conversations at the Design Sprint resulted in an initial idea of simplifying this so that all fields are stored in 'per field' tables. This will greatly simplify the code, making it easier to get it into core. The hope was that the caching model would alleviate most of the performance problems that might result. But there is still a performance concern with this model.

Another approach to this would be to go ahead and store the basic data in simple 'per field' tables, as we discussed, but instead of storing a serialized array in the cache, create multiple tables at an intermediate level that can be queried and filtered easily without any need for joining data. Those intermediate tables will contain duplicates of the basic data stored in the 'per field' tables, but the cost of keeping that data up to date is probably less than the cost of complex joined queries.

If we did that we would no longer store the serialized data in the cache, so the cache_content table would not be needed. Once fields are moved to core, these intermediate tables would probably also eliminate the need for the revisions table.

The new data structure might look like the following. The idea is that each node type would have a table with all its fields and each node might have multiple rows in the table to represent all the delta values of all its fields. Any field could potentially contain multiple values, so any node could potentially return multiple rows and node handling would need to take that into account. But these tables would be easily queriable for any of their values without any joins.

Note that the idea of adding an intermediate level of tables is Dries' idea, this implementation of this idea may or may not be anything like what he was thinking :)

INTERMEDIATE LEVEL

table: node_page

nid	vid	delta	uid	body	teaser	field_phone_value	field_file_fid
1	1	0	1	xxx	xxx	555-5555	34
1	1	1	NULL	NULL	NULL	777-7777	NULL
2	2	0	1	xxx	xxx	555-5555	46

table: node_story

nid	vid	delta	uid	body	teaser	field_name_value	field_nodereference_nid
1	1	0	1	xxx	xxx	xxx	34
1	1	1	NULL	NULL	NULL	xxx	NULL
2	2	0	1	xxx	xxx	xxx	46

BASIC LEVEL

table: field_name

nid	vid	delta	field_name_value
1	1	0	xxx
1	1	0	xxx

table: field_phone

nid	vid	delta	field_phone_value
1	1	0	xxx
1	1	0	xxx

table: field_city

nid	vid	delta	field_city_value
1	1	0	xxx
1	1	0	xxx

table: field_file

nid	vid	delta	field_file_fid
1	1	0	34
1	1	0	46

table: field_nodereference

nid	vid	delta	field_nodereference_nid
1	1	0	46
1	1	0	45

Data Architecture Design Sprint

Final Report

Crell@drupal.org — Wed, 27 Feb 2008 07:54:33 +0000

OK, so, it took a lot longer than I expected or planned, but here is the final summary report from the design sprint. I've provided it in both OpenDoc and PDF format. Hopefully there aren't too many spelling errors. :-)

Data Architecture Design Sprint

The Future of Fields, Drupalcon Boston 2008

bjaspan — Fri, 22 Feb 2008 01:14:40 +0000

This wiki page is for the development of the Drupalcon session we'll be presenting in Boston. We have 90 minutes on Tuesday morning; see our session page.

Please edit this page only if you attended the Design Sprint and will be participating in the session. Otherwise, please give us your feedback via comments. Thanks!

Session outline

Each section to be a presentation with a pause for questions before moving on to next section. The early sessions should be pretty quick because most of it is review of concepts already long-discussed within the community.

Presenters are bjaspan, crell, karenS, nedjo. I (bjaspan) have assigned specific presenters to each section; each presenter will create the slides for their own sections. If anyone objects to my suggestions or does not have time to create the slides this week, just speak up. Please create plain, unstyled slides. I'll merge them all together for simplicity during the session (we don't want to have to switch computers, etc., mid-session).

Overview (bjaspan; 3 minutes)

Idea for and participants in design sprint
Goals and outline of session
- Goal: to present results of design sprint and get community feedback and collaboration

Need (bjaspan; 5 minutes: 3 minutes presentation, 2 minutes questions)

Our vision of Drupal's future
- Web application development platform; CMS is just one in-the-box app
- API driven, not form-driven
- Able to consume and provide web services
- Able to seamlessly leverage the power of contrib for local and web service data
Problems and challenges at present
- Multiplicity of uneven APIs for object handling (user vs. node, etc.)
- Binding to user input (form) model
- Single assumed data store (local SQL DB)

Analysis (nedjo; 5 minutes: 3 minutes presentation, 2 minutes questions)

Some steps to achieve the vision
- Data API: Consistent methods for using local and remote data
- Data rendering: Ability to output in many formats, not just HTML
CCK fields as model
- Already exemplifies much of what we are aiming for
- Consistent method for attaching data to nodes
- Escape the current garbage heap of $node
- Represents a shift from objects or entities (nodes, users) to their fields as the core focus

Initial goal: minimal CCK-based fields in core (karenS; 15 minutes: 10 minutes presentation, 5 minutes questions)

First step: Fields in core
- Preliminary work in D6
- Separate directories
- Split out API from UI
- API in new Fields module, developed in contrib with aim of core
- UI remains in contrib
- Code-level methods for what is now UI-based
- Still limited to nodes
- A small number of sample fields converted
Next steps
- Further implementations (convert more core fields)

Further steps: entity models (bjaspan and crell; 15 minutes: 10 minutes presentation, 5 minutes questions)

Entity model, providing unified operations for all first-level objects (users, nodes, etc.). Two ideas of how this would work. Begin with what the two have in common, then review each for differences.
- model 1
- model 2

Further steps: external data store support (nedjo; 5 minutes: 3 minutes presentation, 2 minutes questions)

Existing Services module. Adapt for core?
Introduce a parallel Clients implementation?

Concrete tasks and how to help (karenS; 5 minutes)

Contributing to CCK adaptation
Work on Services and Clients

Discussion (panel; 30 minutes)

Data Architecture Design Sprint

Proposed Web Service Clients module: Preparing the way for external fields

nedjo — Wed, 13 Feb 2008 20:02:05 +0000

At the recent Data Architecture Design Sprint, participants outlined a development path in which fields can be local or external. A given entity instance (node, user, etc.) can be fully local, fully external, or a combination of the two. For example, a node can have several fields stored and handled in a local SQL database and other fields both loaded from and written to an external SOAP service.

As steps on this path, we outlined (a) short term changes to CCK, (b) a minimal patch to integrate CCK into core, and (c) some follow-up steps, like deciding on a consistent data structure for all entity types, to be followed by a unified set of methods for all entity types (e.g, drupal_save($entity_type, $entity)).

Are there short term steps we can (should?) take now toward external data source support? Here is a rough, conceptual attempt to sketch out such steps in the form of a proposed contrib module, Web Service Clients, taking the existing Services module as a model to work from.

Services module: a model for external data source support

In D5 contrib, a services module was developed to enable standardized development of Drupal as a web services platform. Documentation: http://drupal.org/node/109782, project: http://drupal.org/project/services. Services implements a pluggable architecture in which various types of servers (XMLRPC, SOAP) can be implemented, drawing on a common set of services. Services here are sets of available methods, e.g., node load and user save. Through services, a Drupal site can respond to external requests, providing read/write access to locally stored data.

Clients module: web service clients modelled on Services

Services provides external read/write access to locally stored data. We need the parallel set of implementations: local read/write access to external data sources (via web services).

Parallel to the Services module's pluggable server, we need a set of pluggable server types--but as clients rather than servers. In short, we need a Web Service Clients module (Clients for short).

A minimal initial target spec for this module might be:

Using XMLRPC (the one server plugin bundled with Services module; there are others in contrib), implement a client such that a Drupal site running Clients can meet the following use cases:
1. User can create and update a node that has two fields from an external site running Services' XMLRPC server. On create and update, those fields are not saved locally but instead are saved to the remote service. On load, fields are loaded from the remote service.
2. Whenever a user record is updated locally, a call is sent to a remote service notifying of the update operation. This enables a remote service to store/track a user's ID and associated email address on the Drupal instance.

Solution components:

Generic handlers (in clients.module): shared methods
Pluggable services: services supported, e.g., XMLRPC
Pluggable methods: local methods for handling data operations
Client instances: data unique to specific client instances

Generic handlers

Generic handlers and methods, including:

Connecting to external services.
UI for admin-defined client instances.

Pluggable services

Example: generic SOAP support.

Each service includes methods for:

authentication
deriving schema information about external data sources

Pluggable methods

Define in metadata form a common set of local methods and their expected data formats and return values. Examples:

entity_find: given an array of properties and values, return an array of matching entity IDs.
entity_load: given an entity id, return an entity instance
entity_save: given an entity instance, save to the remote service and return a result code
field_load: given an entity id and a field name, return the value of that field for the given entity
field_save: given an entity id, field name, and field value, save the field to that entity and return a result code

Alternately, we could implement the same set of methods as are available in Services module (see the services plugins), but these may tend to be too specific to the Drupal environment and so to map poorly to external web services.

Client instances

Example: SOAP client to a particular SOAP service.

A client instance may override generic methods for its service type.

Each client instance includes handling and methods for:

Mapping available external request methods to the set of local methods. For example, fed a field_load call, translate this into a call to save data to a remote SOAP service. (Each client instance will implement only a subset of the available local methods.)

Nodeapi and CCK integration

To locally present and edit data from external sources, we need the ability to represent and integrate those data locally.

CCK provides our richest set of applicable tools, and is in any case our proposed long term solution. How therefore can we leverage CCK fields in 5.x or 6.x to provide handling for external fields?

Proposed implementation:

One-to-one mapping between external data fields and local CCK fields, with the CCK field handling overridden for CRUD operations.

For each external field, site admin defines a CCK field and attaches it to one or more content types.
Clients module provides additional UI element in field definition to select external field to map this field to.
In hook_nodeapi() load op, Clients module overrides CCK field handling, inserting externally-derived data in place of local ones (which will be empty, as there are no locally stored data).
In hook_nodeapi() insert, update, and delete ops, Clients module dispatches calls to external web service for fields it is responsible for.

Actions

As well as field-level handling, we need methods to invoke actions based on given state changes. In the above use case #2, this included responding to a user save operation by sending a notification to an external service.

Here we can build off of Actions, particularly the hook operation support in D6.

Client method implementations (particular local methods, e.g., entity_save, that are implemented by a specific client, e.g., a Gazetteer SOAP service) may declare the 'hooks' operations they support, using the same format as used for actions metadata.
These methods are programmatically exposed as actions with associated hooks, to be called as appropriate, receiving in argument form the relevant data structure (e.g., a user object).

Summary

By developing genericized, fields-aware web service client support, we can both provide short term functionality for D5 and D6 and begin concretely to map out long term solutions for core.

After getting our minimal CCK fields into core, we could turn to minimally bringing the Services module and (assuming some solid development by then) accompanying Clients module into core as well.

Does this approach seem to make sense? What considerations are missing? Please comment on or edit up this Wiki page.

[Note: I'm hoping to be able to begin coding a draft Clients module within a week or two for an upcoming client contract.--Nedjo]

Services

Proposed Content Model 2

Crell@drupal.org — Sun, 10 Feb 2008 22:21:19 +0000

During the Data Architecture Design Sprint (or DADS, as I will henceforth call it), we discussed two general models for reaching our vision of "data anywhere, value-add anything". Those models became dubbed "Model 1" and "Model 2", because that's the order in which we happened to write them down.

Barry has already done a good job of explaining Model 1 in an earlier post. Here, I am going to lay out the structure of Model 2. I suspect that the final system, whever it looks like, will draw heavily from both models.

As Barry described in Model 1 (and will probably be repeated in every summary we write for the next few months), Drupal's secret weapon is contrib. Once a piece of data can be made into a node, it automatically gets hundreds of extra features by magic. CCK and Views are the most common and most powerful magic, but by no means the only. However, shoe-horning everything into nodes breaks very quickly when dealing with content that doesn't fit into the "node" model neatly. In core, we have users. In my day job, I frequently have museum artwork databases. I have also considered ways to expose a DocBook tree to Drupal, such as the documentation Steven Peck has been writing for Drupal 6. I will use all of these examples in the descriptions that follow.

The Institute of Contemporary Art

This fictional museum is based on the site that, by coincidence, we were launching during the DADS, the Art Institute of Chicago. There's much more going on with that site, but for now we'll focus on the data access. (If you want to hear more, we've proposed a session for DrupalCon Boston. Go vote for it. :-) ) For consistency, though, I will continue to call it the ICA.

The ICA has their entire archive in a legacy database system. That system has a read-only SOAP interface, using RPC-style calls that return an array structure. There are several types of "thingies" that the system exposes to us: Artworks, Resources, Artists, and Categories. Each has its own integer key-space, in addition to between 2 and 30 single-value fields, ranging from an artwork title to the URL of the large-sized jpg of an artwork. Resources also have a type, which is one of "quicktime", "full HTML", "audio clip", "jpg", etc. Each Resource type needs somewhat different theming treatment.

In Drupal 5/6, the only options for integrating such a 3rd party system are to lazy-create a stub node that wraps a 3rd party object (which can be a bit messy) or to create a completely parallel system that doesn't touch nodes (or all of their yummy value-add) at all. Neither is a great solution.

Entities

In order to be able to value-add to non-local content, we need to be able to define content that is not nodes but still gets node-esque contrb-magic-connection-points. Rather than make everything a node, we generalize those connection points to a higher-level concept formerly known as Thingie and now represented by the name "Entity" (or Prince, if you prefer).

An Entity is a common interface that value-add modules can rely on. Specifically, an Entity type has, and is defined by:

A unique key-space. Users, Nodes, Artworks, and Resource all have a separate key-space. That is, their IDs are not mutually exclusive. User 123, Node 123, Artwork 123, and Resource 123 can all co-exist peacefully.
A set of properties. We discussed "properties" on and off during the DADs, and didn't qutie nail down what they were. The best definition we came up with was "single-value data, and defined by the Entity type rather than the sub-type".
Some set of behaviors. An Entity type is able to define extra logic that applies to it, but doesn't apply to other Entity types. For instance, the User entity type defines the logic relating to logging in a user, a concept that has no meaning for a Node or an Artist.
One or more subtypes. A subtype defines a set of field definitions that it will contain.

Not all Entity types will have or need subtypes. Nodes do now. We discussed possible subtypes for Users, and came up with possible use-cases (users authenticated by different systems, such as local or LDAP) but nothing firm. Artworks and Artists do not have subtypes, however, Resources do. For simplicity, an Entity type that doesn't need subtypes simply has a single subtype, as "1" can always be treated as a special case of "many".

Absent from the Entity definition is whether its fields are local or remote; we established fairly early on that question to be a per-field question. The keys-space of the Entity type, however, may be locally or remotely controlled.

Contrib modules then value-add to an Entity by adding a field to one or more of the types of that Entity. One could add, say, a fivestar field to the Artwork Entity's only subtype, the Resource Entity's "quicktime" subtype, and the "story" subtype of the Node entity.

Some examples

I had originally intended to have drupal_load() always take an integer ID, to normalize it across the board, but I realized while reading Barry's Model 1 description that won't work if the entity has yet to be locally linked from a remote service. That gives an Entity two IDs:

id: The unique key in the entity's native system. In the case of Nodes and Users, it is nid or vid. For, say, a DocBook data source, it could be an XPath query string or the ID of a specific element.
localId: This key, if present, is always an integer and is used by locally-stored fields to reference the Entity. In some cases it will map 1:1 to the id (pretty much any case where the id is an integer, including nid and vid). That is an implementation detail left to each Entity to determine.

Perhaps some code will explain the structure better.

<?php
function node_view() {
  $node = drupal_load('node', 123);
  return "<h2>". $node->title() ."</h2>". $node->field('body')->value();
}
function drupal_load($entity_type, $id) {
  return new $entity_type($id);
}
class Node {
  protected $_id;
  protected $_properties;
  protected $_fields;

Things to note:

Yes, it is OO. OOP is very good at enforcing an interface and hiding implementation details, which is what we want to do here.
title() is a method rather than a direct property. Rendering a human-readable string to represent an Entity is a behavior, and therefore is Entity-type specific.
All properties of the Node class are prefixed with a _ as well as made protected. That way, we can still arbitrarily add extra properties to the node without colliding with them to support older code, at least temporarily until we figure out how to make "everything a field".
Because we're behind an object interface, we can implement lazy-loading or not, or other optimizations, in the constructor, the field() and fields() methods, and in the yet-to-be-written get_fields_associated_with_type() routine. In some cases, lazy-loading makes a lot of sense. In others, such as the Artwork example above, it's more efficient to pull the entire data object at once because the backend in this case supports it. That's information that the caller (node_view()) shouldn't have to care about.
Somewhere in there it would be wise to implement at least static load caching, probably in drupal_load(), but we'll deal with that later.

For comparison, the Artwork Entity might look like this, if we had to deal with non-integer ids:

<?php
class Artwork {
  protected $_id;
  protected $_properties;
  protected $_fields;
  protected $_additionalFields;

Note here that the title is being generated differently, and we're internally classifying fields by where they came from. Some are saveable, some are not. If we try to save an artwork, a local ID will be created if necessary for our "additional fields" to use. Again, the outside world doesn't care. All it knows is that it asks for an Artwork of ID '123.456/5', adds a fivestar rating to it, and saves the Artwork. And until we call save(), the local database is unaffected.

Still an open question, of course, is what the interface for Field will look like. We've already said we want those to always be multi-value, but beyond that, I'm not sure what they'd look like.

For those who like UML, here's what this model would look like, I think:

Data Architecture Design Sprint

Taxonomy as a field

bjaspan — Sun, 10 Feb 2008 18:12:51 +0000

It's useful to explore how existing core functionality can or cannot be implemented as a field. Doing so may cause us to change how we decide to implement them. In this post I'm talking in terms of nodes but trying to avoid assuming that we are limited to nodes instead of Content objects and that all fields are stored locally in SQL. I'm also mostly just thinking as I go/type; I don't have a specific outcome in mind yet.

Taxonomy is a "multi-valued field" assigned to node types currently via variable settings. The term_node table is effectively a field table: (nid, tid) instead of (nid, delta, tid). One note is that the tid assignments to a nid are unordered whereas field values are ordered. Since the nid,tid order has never mattered before, arbitrarily assigning an order via delta should not matter.

During load, term_node connects to term_data and vocabulary. Observation: Although we wrote down that "drupal_load('node', $nid) will SELECT FROM node + load all fields", drupal_load() probably will not actually load all the data itself directly; instead, it will have to call a taxonomy_field_load() hook. For the specific taxonomy case, perhaps we could encode information in the Schema structure or the field definition that tells drupal_load() to join these other tables. However, consider a field that access remote data (e.g. "Amazon products related to this story" field). drupal_load() can't possibly load that data directly, so we need a field_load() hook, so taxonomy should probably just use it.

term_data contains a term weight which determines the order of tids in the $node->taxonomy array. This is disjoint from the delta column in the term_node field table; the weight is a property of the term, not its assignment to a node. Vocabularies have weights too. taxonomy_field_load() will presumably return an array of field values to be merged into the node. When the node is saved, the taxonomy field table (term_node replacement) will be filled with (nid, delta, tid) but again the deltas will be "arbitrary".

So taxonomy_field_load() turns out to look almost exactly like taxonomy_nodeapi('load'); well, we always knew nodeapi was a pretty powerful model.

Today, $node->taxonomy is filled in as an array mapping tid to an "object" of all the columns of term_data. If taxonomy is a field, then $node->taxonomyfieldname is a field array which means its keys are deltas (in this case, meaninglessly so). $node->taxfieldname[n] is a term_data "object" including the tid. So code that assumes the indexes are tids will need to change to use the tid field directly. Such code exists in taxonomy.module, elsewhere in core, and throughout contrib, probably.

Hmmm. I actually just glossed over something important. If taxonomy is a field type, then multiple fields of type taxonomy can be added to a node. This could be quite useful: "user assigned terms" and "editor assigned terms". However, fields are stored based on their name: $node->usertaxterms, $node->editortaxterms, or something like that. Right now, a node only has one set of terms and it is always available at $node->taxonomy. So if I have a node object I can ask, "what are its terms?" because I know where they are stored. With taxonomy as a field, I do not know where they are stored.

This is not limited to taxonomy; comments are the same way, always at $node->comments but if comments were a field they'd be in $node->commentfieldname.

I think the ability to access $node->taxonomy directly is an historical artifact. Drupal was designed as a CMS and taxonomy, comments, and other CMSy features were built it in fixed ways. Many sites use Drupal for events, and event nodes often have a venue, but we do not expect to be able to access $node->venue to find out the venue for an event node. We have to know the name of the venue field.

What to do about this? Random thoughts:

The easy solution: At install time, when we create the 'page' and 'story' type, we create a field of type taxonomy named 'taxonomy' and assign it to those node types. If you create a new node type and want it to play with standard taxonomy functionality, assign the 'taxonomy' field to it. Otherwise, feel free to do whatever you want.
Perhaps we need to be able to ask a field type for all of the values on a node from that field type. e.g. $taxonomy = taxonomy_field_all_values($node). Or probably, since the core field registry knows what fields are assigned to each node type, it would be $taxonomy = field_all_values_of_field_type('taxonomy', $node). That would give me all the terms for the node. If I specifically wanted some subset (e.g. only one field's worth of terms), that implies I'm writing new code in a post-taxonomy-as-field world and presumably I know which field I'm looking for.
Perhaps what's going on here is that we're trying to turn Drupal into a more general web app development platform and we're discovering that it has aspects of the initial app built on the platform, the CMS app, integrated to deeply into core. The CMS app defines that $node->taxonomy and $node->comment exist as those specific names and then uses them. So perhaps we need to very clearly separate "core Drupal functionality" and "the CMS app" from each other. They can still be shipped in the same tarbar but as clearly defined entities.

All that from "let's think about taxonomy as a field"!

Data Architecture Design Sprint

Summary of the Data Architecture Design Sprint

bjaspan — Sun, 10 Feb 2008 17:17:29 +0000

Six Drupal developers met for three days in Feburary 2008 to work on and re-design Drupal's core data architecture. We began with an overly broad list of topics and goals and spoke in very general terms about "Drupal's overall mission" and similarly vague concepts. By the end of the design sprint, we devised a tangible vision for Drupal's future and an understandable method for getting there.

We believe that the overall goal for the Drupal community should be to convert Drupal from a monolithic Content Management System into a web application development platform in which a powerful CMS is one built-in implementation. We talked a lot about what that means and requires, including things like consistent and documented APIs for all parts of core, switching to an API-driven instead of form-driven architecture, and a data rendering architecture for producing output in many formats besides HTML. None of these are new ideas within the community, of course, but discussing them in the context of our stated goal (making Drupal into a web app development platform) provided a useful organizing structure and new insights.

One particular epiphany we had is the realization of how Drupal's existing architecture makes it an excellent tool for developing web applications in a web-services world. Drupal receives a request from an external source (currently, from a web browser or XML-RPC client), converts it to a query against a data store (currently, an SQL database), and returns the result. Before it returns the result, however, it exposes the data via Drupal's hook mechanism to an enormous variety of contributed modules that can add additional value to the data. The fact that so many disparate people can write modules to enhance content value through Drupal's hook interface is Drupal's key feature.

It became clear that, in a web services world, where Drupal needs to be able to both offer and consume web services, Drupal also needs to be able to provide its key feature---value-added data via contrib modules---to web services data. Comments, votes, maps, social bookmarks, etc. etc. etc., should be just as appliable to a content object retrieved from Amazon as a node retrieved from SQL.

We eventually decided, as we mostly knew we would, that moving CCK's field concept in Drupal core is important step forward. Keeping our two days of discussion about "the future of Drupal" in mind, we worked to define how to put fields in core in a way that is consistent with our proposed future and makes it easier to get there. We devised two "Proposed Content Models" for supporting what we called "multi-source" data with core fields though both need plenty more design work and exploration.

A lot of work still needs to be done and, as always with Drupal, most of it is currently slated to be performed by volunteers. It is our strong desire to at least have node-based fields in D7 core and, hopefully, have multi-source data support in D7 core as well. We will have to see what is achievable.

Data Architecture Design Sprint

Structure of "fields in core" patch and of CCK project

nedjo — Sat, 09 Feb 2008 18:11:24 +0000

Following the design sprint, drawing on what we had discussed, Yves, Karen, and Nedjo tried to map out some of the structure of a core patch and the accompanying short and longer term changes to CCK fields.

Specifically, we discussed the following approach:

In the immediate term, make small changes as feasible to facilitate longer term changes, e.g., move CCK modules into their own directories in CVS.
Working in CCK HEAD or possibly a D6 branch, do as much as possible of the work of preparing the patch for core in the form of refactoring the CCK package. This should be possible because we are preparing a new set of APIs that will be isolated in their own files and won't immediately require large changes to existing Drupal core code. We avoid the work of rolling patch after patch.

Core patch structure

What concretely will a core patch look like?

New required core API module, Field. While initially implemented for nodes, it is abstracted to facilitate later application to other entity types.
Field types
- Could be implemented in a shared module, Field Types, with the various core field types in a single module. Alternately we stick with the current approach, one module per field type.
- Whether one or more modules, do they need to be required? Initial feeling is, many or most may need to be required.
- We could consider UI tweaks to the modules admin page to e.g. collect field type modules together.

Short term CCK steps

Move all CCK modules to their own directories (except content?). Issue: http://drupal.org/node/218973.
Branch 6.x
Likely after branching, create new field module (in its own directory) and move field CRUD API from content module to field module. Content module eventually becomes a purely UI module.
In Field module, implement hook_nodeapi() as is currently done in CCK to implement fields CRUD. When we get to the point of a core patch, this hook implementation will be replaced by calls in node_load(), node_save(), etc.

Data Architecture Design Sprint

Remote content as minimal viable functionality

bjaspan — Thu, 07 Feb 2008 23:42:28 +0000

We talked a lot about minimum viable functionality (MVF) for D7 core. We also talked briefly about what criteria to use to decide what should be in core vs. contrib. I'd like to revisit that.

Question: Why bother putting fields in core at all?

Being in core has many drawbacks. Much slower development/release cycles, necessity to gain broad consensus for everything, requirement to operate in every possible environment that Drupal supports (e.g. low-memory hosting), etc. We already have CCK and fields in contrib. They work great. Why are we going to incur all this extra pain?

I think the answer to "what should be in core" is "those things that provide substantial benefit and simply could not be done outside of core because they require fundamental architectural support or the like."

CCK fields on nodes do not meet this requirement. We already have a 'body' field. Just providing a new way to have a 'body' field is not interesting or useful enough to incur the extra effort.

Instead, I assert that the MVF for fields in core needs to include something that is just not possible with CCK in contrib. Among the things we discussed in our sprint, the ability to add fields to remote content is that thing. Fields on remote content will by itself make D7 a killer release, leaping Drupal past its competition. I think it will make Jeff, Acquia's VP of marketing, drool. I think this is the goal we need to strive for.

I have a basic grasp of how to implement my proposed Content Model 1 and I do not think it will be a monster task; most node code and functionality will remain the same. I still want to see Larry's writeup of Content Model 2. Then I/we should get to serious work on nailing down the Content interface (or Fieldable as I've suggested renaming it) or whatever Model 2 uses.

Thoughts?

Data Architecture Design Sprint

Day 3

chx — Thu, 07 Feb 2008 01:28:12 +0000

Notes: Comments are now enabled on this page.

CCK simplifications for core

The day started with ways we can simplify current CCK functionality because there is a feeling that what we have is way to fragile and complex for core at this time. Functionality slated for destruction includes:

Per-content type field storage, e.g. no more content_type_story tables with columns for multiple single-value fields. All fields are stored in today's equivalent of content_field_fieldname tables.
All field tables always have a delta column even if it is a single-value field.
Field-level settings, which are defined as those that (at least) affect the schema, cannot be changed. e.g. You cannot convert a text field from plain-text to formatted-text; instead, you must create a new field, copy the data, and drop the old one.

The net effect of these changes is that field tables never need to change dynamically except during version upgrades. No other part of core performs dynamic schema changes outside of update and it we felt it was much simpler to maintain this restriction.
bjaspan: I suppose there is an argument that if we are removing the ability to change field settings dynamically, allowing per-content-type storage and per-field storage does not imply/require the same level of complexity currently in CCK because if you can't change the field settings you can't change the storage method so we can still lose all the code that handles changing the storage method.

Field and Field Instance Definitions:

A field type is a set of functionality implemented by a module, e.g. text, nodereference, gmap.
A field is a collection of settings of a field type: e.g. a text field, max length 60, plain text (no input format option for the user).
A field instance is the binding of a field to a content type with additional settings: e.g. the above text field assigned to the story node type with the label "Five word summary."

The field settings contain aspects of the field that will be the same in all instances. They often include things that affect the database schema or storage and the way the field data will be stored. Because of this, they cannot vary from one instance to another or you would have a field with a different database schema. Examples of field settings are the size of the field and whether it can hold multiple values. (bjaspan: Since all field tables will always have a delta column, I'm not clear on why single/multiple cannot be an instance setting UNLESS we continue to allow per-content-type storage as discussed above, though clearly changing a field from multiple to single will lose data).

The field instance settings contain aspects of the field that can be different from one instance to another. They include things that do not affect the way the data is stored, only the way it is displayed. Examples of field instance settings include the label and weight of the field.

The settings are stored in two tables, one for the field settings (one row for each field) and another for the field instances (one row for each instance).

Open questions

Is it necessary to have two separate tables or could all the data should be stored in the instance table, even if the instances have duplicate field information. (bjaspan: That would be non-normal form and we'd need a good reason for is; is there one?)

Should field settings be locked, once created, so they cannot be changed? They impact the schema and it would greatly simplify core code if that is just not allowed. It would be OK to change instance settings like labels, but not field names, sizes, multiple, and other things that impact the database schema.

Multiple Values

We discussed whether core will need to handle multiple values and concluded that will certainly be necessary. So we examined the way that the D6 version of CCK handles multiple values to start thinking about what will need to be implemented in core.

The D5 and earlier versions of CCK uses a separate formatter for each individual field value, one after the other. That does not always make sense. There should be a possibility to format all values for a field in a single formatter (like a field that contains multiple points on a map or a slideshow of images). This change has gone into the D6 version of the code.

The D5 and earlier versions of CCK presume widgets will handle multiple value forms themselves and pass them back to the Content module. In D6 that is changed so that widgets create a single form value element and it is the Content module that combines them together into a single form element. Some widgets need to opt out of that behavior to handle all values in a single form element (like a select list or checkboxes). Now that the Content module handles multiple values, there is also a way for the widget to opt out of that and handle its own multiple values. The optionwidgets module does this.

API

Example code

What will the API look like. An example of what a module developer might write:

$info creates a content type called 'user_profile', it contains stuff like 'name', 'has_body', 'min_words', etc. (except that body probably won't be a built-in part of node types anymore, it will be a normal field).
node_type_save('user_profile', $info);

$settings = array(
'name' => 'Eye Color',
'restricted values' => array(
'brown', 'green', 'blue'));

field_create_field('eyecolor', $settings);

$settings = array(
'label' => 'User eye color');

field_create_instance('user_profile', 'eyecolor', $settings);

To add this field to another content type:

$info creates a content type called 'admin_profile'.
node_type_save('admin_profile', $info);

$settings = array(
'label' => 'Administrator eye color');

field_create_instance('admin_profile', 'eyecolor', $settings);

Needed Functionality

Create node type / Delete node type
Create field / Delete field
Add field to node type / Delete field from node type
Update field configuration (required, multiple, widgets/formatters, weight)
Get table name for field
Get types
Get fields
Get field instances for type

Open questions :

Current CCK field CRUD API operate only on 'field instances', and do not allow fields without instances. What sense does it have ? Is that even doable (both Karen and Eaton came to the conclusion that it might be a problem)? Karen did not remember what the problem was; we are thinking that the simplifications we are planning will eliminate it.
Other drupal CRUD APIs merge 'create' and 'update' into a 'save' function...

Approaches for letting modules define node types and their fields

imperative : sequence of direct field CRUD API calls at install time
declarative : cf hook_node_info(), hook_schema()
-- modules provide returns an array describing node types and their fields
-- this array gets processed and turned into a series of field CRUD API calls.
-- the hook_update_N() mechanism calls API functions for updates is used for version updates

We decided that we need to implement the API in either case so we're starting with that. A declarative approach (hook_node_types() + hook_update_N() for changes) is more consistent with other parts of Drupal and can be added on later.

Workflow

Typical workflow for module authors (Just like what currently exists for Views) :

Adjust their fields setup using contrib CCK-UI
'Export' types and fields in a PHP definition (new 'Content Copy')
paste the PHP snippet into their hook_node_info (or whatever the actual hook)

Open questions :

UI-defined fields probably need to live in a different namespace (see below), so this workflow probably hits name issues.

Node structure :

$node->fieldname[n]['column']

Field namespace :
It's the job of the modules not to create fields whose name clashes with other modules' fields.
We need to ensure that module-defined fields don't clash with user-defined fields (with CCK UI in contrib)

What happens when disabling the module that originated the field_create_field ?
- skip field
- raise exception

Compatibility with existing module

The 'core fields' won't be able to co-exist with a D7 forward-port of current contrib CCK, if only because there can only be one 'text.module'. Thus, we need to get 'fields in core' right :-)

Node Load

Current node load operations (e.g. loading a poll node):
1) node_load: SELECT FROM node, node_revision
2) hook_load: SELECT FROM poll
3) hook_nodeapi op load: SELECT FROM comment

New :
1) node_load: SELECT FROM node, node_revision and load all fields.
2) hook_load: calls poll_load() so the poll module can do something if it wants but it does not need to load its field-based data
3) hook_nodeapi op load: SELECT FROM comment, for any modules that are not converted to fields (e.g. if comments are fields, SELECT FROM comment is handled automatically in step 1).

If we go to a class Node, loading of all fields in step 1 can actually be lazy-loaded so unused fields are never actually loaded/processed.

Which CCK functions should go to core ?

Go to Core:

field storage engine
Field CRUD
widgets / form gen.
Node CRUD
formatters
add more / drag 'n drop re-ordering
Field validation
custom multiple value handling

Stay in contrib:

field default value
field allowed values
Field CRUD UI (Manage Fields tab)
Display UI (Display Fields tab)
Fieldgroups
Content Copy
3rd party Drupal integration (Views, Pathauto, Token)

Which CCK fields should go to core?

The current CCK core fields:

field types
fieldgroup
noderefence
number
userreference
image
file
computed
date
text

widget

optionwidgets

Things in core that might be fields :

body: text field
created: date
upload: file
user pic: image
IDs: number+options

Criteria for 'in core' :

wide applicability
minimum useful functionality

Action plan:
Start with text, number, optionwidgets, image, file. Leave nodereference and userreference in contrib until we need them in core so we can continue to easily add features and do other work on them. Once in core they will be hard to change. Getting a date field into core would be easier if improved date handling gets into core, which is a separate initiative. The complex date field would probably remain in contrib but a simpler date field could be created for core.

Next Steps

Beyond a minimal implementation of CCK fields in core, what will be the next steps?

node as class?
- To get a consistent set of methods for all first-class entity types (node, user, etc.), we need a consistent data format--array or class. If we use class, we may find it advantageous to use a custom rather than StdClass to have access to PHP 5 features such as lazy loading.
gradual conversion of existing core 'fields'
- Having implemented a small number of new-style fields in core, we will need gradually to convert much (all?) of the remaining existing 'fields' to the CCK-style system. Doing so will require new methods and functionality. For example, existing CCK cannot fully handle the current node taxonomy implementation.
field property can not be edited

Lessons learned for future 'Design sprints' :

A facilitator was very useful. We benefited from having a process oriented faciliator who was a little removed from the issue (Nedjo).
The initial scope too large. We did narrow it, but not in time, so we prepared for things that weren't directly relevant.
Sometimes key contributors need financing to attend. In spite of the cost it would be a worthwhile investment output.
Commercial community understands the value, but employee time is precious.
The final number of six people was good size.
Eight hours a day is good. It's tempting to go longer, but wouldn't have been a good idea.
The nightly wiki summaries was a useful way for participants to analyze and review the results.
Taking complete notes is critical, could it be done by someone else (e.g. a support role)? It would have to be someone who understands the discussion.
Fun is good, eg. dinner out. There is a lot of pressure and intensity and providing some respite helps the process.
Three days feels like an appropriate length of time.
It is useful to have people on a similar skill/expertise level
- invite key contributors for task on hand
- also useful to have a mix of the centrally involved with in this (some use CCK others not.)
Getting everyone face to face in a small group really benefit the process.
The facility needs
- whiteboard
- wifi
- sufficient space
A nice location (Hawaii) would be nice! :-) Chicago in Feburary not so good.
Put everyone up at the same hotel. It avoids wasting time and provides more chances for unscheduled interaction. Lots of good ideas came up while eating at Panera!

Data Architecture Design Sprint