Fields in Core

This Wiki page summarizes the UI needs for the D7 Field API, to get the ball rolling on design discussions.
Discussion currently takes place in the comments here - or is there any other place more in line with D7UX habits ?

Introduction

A good UI for D7 Field API remains to be invented - esp. when considering a couple new features compared to CCK D6.

With Field API in Core I think we are shooting ourselves in the foot if we ship without a UI for it.

That having been said, let's see where we are at.
The Current Field UI / CCK UI still lives in contrib because it is said to be too complex or downright bad to belong in core.
Hence, if we want a Field UI in core we need to make it less complex. If you visit admin/build/types with CCK UI enabled you have have an admin page that needs to achieve all of the following:

1. Creating content types
2. Edit existing content types' settings

The objectives of the sprint were to flesh out a lightweight basic design for integrating RDF into Drupal core and implement as much as possible the features we agree on during the sprint.

In http://drupal.org/node/367595 the groundwork is being laid for multilingual support in the core fields api.

At a May 21, 2009 IRC meeting to plan translatable fields (see this summary), a consensus emerged that translatable fields should emulate the current Translation module behavior.

With translatable fields we are introducing a second way in which nodes (and other Drupal data objects) can be made translatable, alongside the existing (node-only) one.

The purpose of this wiki page is to begin to map out in detail just how field-based translation will work, taking as our basis the existing node-based translation. For each feature of node-based translation, will there be an equivalent in field-based translation? If so, what will that equivalent look like, and how will it be represented?

Most or all of this will go as follow-up patches to http://drupal.org/node/367595; but it's worth starting to map the work out now.

This page so far has a few preliminary notes and needs a lot of expansion. Please add and edit. What further aspects of the existing node-based translation will we need to emulate in field-based translation? What will we need to consider?

Three key functions:

From rdf.api.php in the development bzr repository we get definitions for our two hook functions:

<?php
/**
* Allow modules to define their own RDF mappings.
*
* @return
* An associative array of mappings. Most keys will be bundle names, but
* mappings can also be defined for non-bundles.
*/
function hook_rdf_mapping() {
return array(

To become an update for the issue RDFa: Add semantics from the ground up:

UPDATE ON THE UPDATE: This is the approach we think could be taken in contrib for RDF, and the original title Proposal for RDFa in Core, storing and loading, after first day of sprint has been changed. You can see the current approach over here.

Our CRUD functions:
<?php
rdf_mapping_create($bundle, $attribute, $rdf_term);
rdf_mapping_read($bundle, $attribute = NULL);

There are only a few months left before the code freeze on September 1st. Now that Fields API has settled in core, it's time to extend it with some RDF semantics. DERI Galway is hosting an RDF in Drupal code sprint during the week of May 11th until May 14th.

Goals of the code sprint

The RDF code sprint will focus on Drupal core and aim at integrating RDF semantics in it.

1. Extend Fields API to integrate RDF mappings for each field instance. The semantics of a field can differ from a bundle to another. This can be stored either in the existing settings property or by adding a rdf_mappings property to the Field Instance objects.
2. Modify the Fields UI (contrib) to allow RDF mappings editing.
3. Define the appropriate mappings for the core modules, based on the RDF core mapping proposal.
4. Patch core modules with the mappings defined above.
5. Export these mappings in RDFa via the theme layer and keep it as generic as possible in order to ease the work of the themers.
6. Write tests for RDF in core.
7. Identify other non-fieldable entities in core which could benefit from being RDF-ized, and see how to annotate them. Comment is one example. Terms also, though they might become fieldable.
8. RSS 1 (RDF) in core. Arto volunteered to get started with that.

I've been experimenting with implementing complex fields by creating a field type that's also a fieldable entity.

But I've run into some problems in the form department. The issue is that $form['#fields'] makes it rather difficult to include a field in a complex element, when field implementations expect the form global #fields to include the the field and instance. It's 'just' form API #process functions that seems to be the problem.

Has anyone talked about devolving title, body(for nodes) and mail and name for users into fields

One reason for this would be to have mail and name be non-unique if a setup called for that. See this comment.
http://drupal.org/node/394282#comment-1408924

If anyone has discussed this somewhere let me know?

I've been thinking about the concerns about storing fields in separate tables and have another idea to knock around. Storing each simple field in its own table could obviously result in a lot of tables for each content type, but storing a few complex fields could significantly reduce the number of tables involved. So another option is to make it easier to create a complex fields to match the needs of a particular system. If that was easier to do, anyone concerned about performance could combine the fields they need into a smaller number (or even a single) complex field.

Today I mostly goofed off at work and created the Per-Bundle Storage module (pbs.module), a Field API contrib module that stores all values for all limited-value fields in a bundle in a single table row. Per-bundle storage is the exact analog of pre-Field API CCK's per-content-type storage and thus provides the same benefits (namely, loading all such fields in a single query).

We currently have three Field API groups: CRUD, Type, and Attach. As recently as Monday, chx was confused about what these names mean; he thought "Attach" was for attaching a field to a bundle. If he can be confused, then the names are just confusing. I propose that we change them as follows:

We have a task for cleaning up the serialized data in the field_config_instance table. This includes preventing duplicate/inconsistent data from existing in the serialized data column and other columns, and considering whether some data now in the serialized column that did not used to be should in fact go back into its own column.

1. Preventing duplicate/inconsistent data

In _field_write_instance, we previously wrote $instance (the whole record) into the 'data' column. I changed the code like this:

<?php
// Create $data to contain everything from $instance that does not

Current code is full of t($instance['label']) as a CCK legacy.
I know that this is not the recommended way to make user-defined strings translatable, but I'm not sure what the right way is.

Taken from email discussion - Quick summary :

chx :

yched : This doesn't work because for this forbids reference-altering the arguments

chx: Then let's use the 1st form when by-ref args are needed and the 2nd one for the rest.

(end of summary)

I think we need another Field Attach API function.

Currently, a fieldable entity calls field_attach_form($obj_type, $obj, $form, $form_state) to load the appropriate field elements into $form for editing $obj. Later, it calls field_attach_validate(..., $form, $form_state) so the field widgets can perform their validation. Eventually, it will call field_attach_insert($obj_type, $obj) or field_attach_update($obj_type, $obj) to store the field data in the database.

We have a Field CRUD API task: "CRUD: API to load single field for single entity: Should take field, entity, and entity_id as arguments. Needed for Materialized Views."

I started to write this function, thinking it would be easy (ha). I decided to make it exactly like field_attach_load() except for a single object and with another argument for the single field to load, like this:

field_attach_delete() deletes all fields for the specified object from the current and archive field data tables.

During the code sprint it was decided that core would not allow field attributes to be changed, only instance attributes. This was to prevent destructive changes, like trying to change a field type. But not allowing field changes creates another list of issues. Yves and I talked about this some last Friday (the last day of the sprint that got aborted by bad weather in Boston). Here are my (rough, unedited) notes from that discussion:

No field update:

Several kinds of fields currently offer an 'allowed values' list where you can create either a hard-code list of possible value or some PHP code that will produce an allowed values list. The allowed values list is a field attribute that is shared between all instances of a field.

Field API does not yet support revisions. It can easily enough, but we have a few decisions to make.

In D6 CCK, all per-field and per-content type storage tables contain columns for nid and vid but the primary key is vid. Multiple versions all go into the same table.

We made a lot of progress during the Field API code sprint but the code is by no means "finished" for D7. Our planning spreadsheet shows many tasks remaining to be done, though not all of them are ever intended for core in the D7 timeframe. And, of course, we have yet to convince the community that our approach (particularly the per-field storage tables) is correct. So the code is in no way core-worthy yet.

It's been a crazy and exciting week. We grappled with a lot of issues, went around in plenty of circles, and took some heat from the community. We did not accomplish nearly everything we wanted to but did achieve several important goals. The big one is that it is now pretty simple to "field-enable" any kind of object with minimal effort. We have field-enabled nodes and users.

There have been quite a few posts to the Fields in Core group about the decision to put all fields in individual tables, despite the apparent scalability penalty of removing the per-content-type (per-"bundle" in the Drupal 7 Field API jargon) tables. The reason is that we have something better to replace it with, and we've decided to simplify the Field API code by removing now-unnecessary code related to per-content-type tables.

Pictures from the sprint
Participants: Dries, Karen, Yves, Moshe, Barry, David Strauss, David Rothstein(1 day), Karoly, Florian
Current status from the Fields in Core sprint, via chx in #drupal

Database first:

Storage will be per-field.

What's a field?

We considered preserving support for table-per-content type. There are a lot of reasons why we wouldn't want to, and one of the primary objections to abandoning them is the performance penalty of increased JOINs. So, I decided to run some benchmarks.

I set up a benchmark environment:
* 64-bit RHEL 5
** Dual quad-core Xeon
** 8GB RAM
** RAID-1 10K RPM SAS
** PHP 5.2 with APC
** MySQL 5.0 (InnoDB)
** Apache 2.2
* Drupal 6.8
* CCK 6.x-2.1
* Devel 6.x-1.12
* Views 6.x-2.1

I set up a content type with the following fields (aside from included files):
* Text (single, unshared)

Getting a working copy

• Download and install Bazaar. Packages are available for many platforms.
• Get a working copy: "bzr branch bzr://vcs.fourkitchens.com/drupal/7-fic my-fic-working-copy"

While you're working

• Run commits (bzr commit -m "My message") to checkpoint your work. These commits only affect your local working copy.

Updating your working copy to get the latest Fields in Core code

edited to add this was posted by David Strauss

My primary interest in having fields in code is the ability to retarget our node storage back-end. (And when I say "nodes," I really mean any object using the fields API.) Relational databases -- at least the open-source ones -- make a poor storage engine for nodes.

This document is the (evolving) plan for the Fields in Core Code Sprint at Acquia from December 15-19, 2008. This is a DRAFT. Many sections are incomplete. Comments encouraged!

This document has been moved to Google Docs for easier online collaboration. The link is:

To view the document: http://docs.google.com/Doc?id=dcz939d6_2gnh78thd
To edit the document: https://docs.google.com/a/acquia.com/Doc?id=dcz939d6_2gnh78thd

Everyone can view the document. People who have been invited as collaborators can edit it. If you want to edit it or if there is a problem, contact Barry Jaspan (firstname dot lastname at acquia dot com).

Below are the notes I already emailed a few people in the group. Those are my review of the kick-off work Barry did in his SVN, along with thoughts on tasks and open questions. Not all of those have to be adressed / answered before we can move forward, this is more of a reminder list.
I'm posting this here so that it's online somewhere instead of in a few mailboxes.

This page is a wiki, so it can be freely edited. Formatting gave me a hard time, there might still be a few glitches here and there.

Good news! During the week of December 15, we're organizing a 5-day Fields in Drupal core code sprint at Acquia! The goal is to get CCK functionality into Drupal 7.

So far, Karen, Yves and Barry have signed up -- Karen and Yves are the main CCK maintainers, and Barry has done a lot of work on CCK as well.

To help us fund the sprint, please consider making a donation using the ChipIn widget on this page. We need money for airline tickets, hotel rooms, food and transportation. It would also be great to fly in a few additional people with extensive core and CCK experience.

I've tentatively worked out a budget of $7,000 USD, which covers flight, food and hotel costs for at least four people (Karen, Yves, and two additional people). Since Acquia is covering my travel expenses and allowing Barry to participate all week long, that gives us six people working on CCK-fields-in-core for an entire week. Any excess money will be used to add more people, or donated to the Drupal Association.

To guarantee that Yves and Karen can attend, Acquia is funding Yves' and Karen's hotel and airplane tickets if enough money can't be raised through donations. Acquia is also providing working space in our Andover office.

We'll try to allow people to participate in the sprint remotely, and provide a daily update on our progress. If you're interested and available to participate, join the Fields in Core group, enable e-mail notifications, and block time in your calendar between December 15 and December 19. We'll use the Fields in Core group to plan and to let you know how you can contribute and participate.

With Drupal 6, menu tokens are able to automatically load database records given an object id, this is a great feature, however this requires developers to implement 'load functions' for loading their records. Those 'load functions' are often just a wrapper around 'SELECT * from {mytable} WHERE mykey = %d'.

We could do better, and allow automatically load operations, and even better, enable automatic basic CRUD.

Please see http://drupal.org/node/265604.

Much has been said (http://groups.drupal.org/node/9297) about how fields should be structured in D7 core, what aspects of fields can be changed, and how those changes are implemented. It is (past) time to move forward on implementing fields in core and in this post I am proposing an answer.

Disclaimer: I make a lot of declarative statements in this post. Obviously I do not have unilateral authority; this is just a proposal.

Field storage

This discussion will revolve around issues of Remote Field Revisions. This regards getting field population from single, multiple, and unknown sources. Sending data to other sites and how they can use, modify, and potentially change that data from source. Also regarding race state and origination of data sources and trusted sources for field revision.

This discussion will revolve around issues of Remote Field Revisions. This regards getting field population from single, multiple, and unknown sources. Sending data to other sites and how they can use, modify, and potentially change that data from source. Also regarding race state and origination of data sources and trusted sources for field revision.

Here are my notes from the Future of Fields presentation from this morning.

[Begins showing a picture of the Data Architecture design Sprint including: chx, yched, bjaspan, crell, karens and nedjo]

Drupalcon used to be a developer conference where they would meet and figure out what was going to happen for the next year, but in Barcelona they realized that it was harder to do that.

So they asked about the who are the right people to be involved, and they all flew into Chicago and did a sprint.

Here are my notes from the Future of Fields presentation from this morning.

[Begins showing a picture of the Data Architecture design Sprint including: chx, yched, bjaspan, crell, karens and nedjo]

Drupalcon used to be a developer conference where they would meet and figure out what was going to happen for the next year, but in Barcelona they realized that it was harder to do that.

So they asked about the who are the right people to be involved, and they all flew into Chicago and did a sprint.

The way that fields are structured in CCK now is that any field that is multiple or shared has its own separate 'per field' table, and all other fields are grouped together in a 'per content type' table. Querying this data to create a node can be expensive, so the serialized node is cached and the cached data is used during node_load().

The way that fields are structured in CCK now is that any field that is multiple or shared has its own separate 'per field' table, and all other fields are grouped together in a 'per content type' table. Querying this data to create a node can be expensive, so the serialized node is cached and the cached data is used during node_load().

OK, so, it took a lot longer than I expected or planned, but here is the final summary report from the design sprint. I've provided it in both OpenDoc and PDF format. Hopefully there aren't too many spelling errors. :-)

OK, so, it took a lot longer than I expected or planned, but here is the final summary report from the design sprint. I've provided it in both OpenDoc and PDF format. Hopefully there aren't too many spelling errors. :-)

This wiki page is for the development of the Drupalcon session we'll be presenting in Boston. We have 90 minutes on Tuesday morning; see our session page.

Please edit this page only if you attended the Design Sprint and will be participating in the session. Otherwise, please give us your feedback via comments. Thanks!

Session outline

This wiki page is for the development of the Drupalcon session we'll be presenting in Boston. We have 90 minutes on Tuesday morning; see our session page.

Please edit this page only if you attended the Design Sprint and will be participating in the session. Otherwise, please give us your feedback via comments. Thanks!

Session outline

At the recent Data Architecture Design Sprint, participants outlined a development path in which fields can be local or external. A given entity instance (node, user, etc.) can be fully local, fully external, or a combination of the two. For example, a node can have several fields stored and handled in a local SQL database and other fields both loaded from and written to an external SOAP service.

At the recent Data Architecture Design Sprint, participants outlined a development path in which fields can be local or external. A given entity instance (node, user, etc.) can be fully local, fully external, or a combination of the two. For example, a node can have several fields stored and handled in a local SQL database and other fields both loaded from and written to an external SOAP service.

During the Data Architecture Design Sprint (or DADS, as I will henceforth call it), we discussed two general models for reaching our vision of "data anywhere, value-add anything". Those models became dubbed "Model 1" and "Model 2", because that's the order in which we happened to write them down.

Barry has already done a good job of explaining Model 1 in an earlier post. Here, I am going to lay out the structure of Model 2. I suspect that the final system, whever it looks like, will draw heavily from both models.

During the Data Architecture Design Sprint (or DADS, as I will henceforth call it), we discussed two general models for reaching our vision of "data anywhere, value-add anything". Those models became dubbed "Model 1" and "Model 2", because that's the order in which we happened to write them down.

Barry has already done a good job of explaining Model 1 in an earlier post. Here, I am going to lay out the structure of Model 2. I suspect that the final system, whever it looks like, will draw heavily from both models.

It's useful to explore how existing core functionality can or cannot be implemented as a field. Doing so may cause us to change how we decide to implement them. In this post I'm talking in terms of nodes but trying to avoid assuming that we are limited to nodes instead of Content objects and that all fields are stored locally in SQL. I'm also mostly just thinking as I go/type; I don't have a specific outcome in mind yet.