Database

Databases: The Next Generation task list

Crell@drupal.org — Sun, 24 Aug 2008 09:54:39 +0000

With the new PDO-based database API now in core (yay!), there are a number of follow-up tasks that need to be taken care of. They were left out of the original patch for a variety of reasons, mostly keeping the already massive patch from becoming even more massive. :-)

The full Drupal 7 database issue queue contains over 50 items that should be addressed at some point, but the following in particular relate directly to DBTNG. Volunteers welcome and encouraged to help out!

http://drupal.org/node/299308 - Explicitly require PDO in the installer. We do kinda require this now.
http://drupal.org/node/298669 - First steps toward restoring database query logging in order to make devel.module work.
http://drupal.org/node/67349 - Support SQLite databases. Really. We want this. A lot. :-)
http://drupal.org/node/195416 - Make database table prefixes per-connection. If you want to use prefixing and multiple database connections, well, right now you can't. Let's make it possible. This does impact SimpleTest, too, but if done properly can probably make simpletest's code easier.
http://drupal.org/node/111011 - Add foreign key support to Schema API. Even though core can't leverage them directly, extra metadata can be useful for PHP-level maintenance. It also is a prerequisite for letting Views leverage Schema API directly.
http://drupal.org/node/299268 - Support "virtual fields" in Schema API.
http://drupal.org/node/299176 - Exterminate db_rewrite_sql(). And there will be much rejoicing.
http://drupal.org/node/299178 - Support subqueries in FROM clauses.
http://drupal.org/node/299267 - Add "extenders" to Select queries to support tablesort, pager, and views.
http://drupal.org/node/299269 - Flag slave-server-safe queries for better SQL replication.
http://drupal.org/node/299433 - Figure out how we want to remap dynamic query operators.

And of course, convert all of core to the new API so we can remove the BC layer. :-) That can and should happen in bits and pieces, as well as part of any new patches from now on.

Database

Table "Node" Field "language"

superjacent — Fri, 20 Jun 2008 04:45:10 +0000

Drupal 6.x. Table "node".

What should the value of the "language" field be if the displayed text is Chinese.

If a value is required, are there any other things that need to be done so that Chinese symbols are correctly displayed.

Any help appreciated.

Steve
http://prime357.org

ps. I've got a thread running at my site which I think this may be a part issue to it. As you can imagine it's becoming very confusing as all I see is either unicode or Chinese symbols (can't read Chinese).

Postgresql

Looking for Contributors - C++ (and other stuff in time)

superjacent — Thu, 19 Jun 2008 11:37:27 +0000

Hope I'm not out of line with this request.

I'm now on the lookout for C++ contributors (and other like stuff in time). I started the site http://prime357.org, which is a separation from my personal blogging (http://superjacent.net - running) site, in early April this year. Too much geeky stuff was creeping into the running site.

The purpose of Prime357 was to track my development in C++ and a place to continue development of the Wordpress to Drupal converter, a Realbasic application. I've got some good how-to type tutorials (my opinion) and short demonstration programs which are presented via the Drupal book method. I first wanted to get some content into the site before seeking contributors so at at least anybody interested could get a feel for my intent (not that it's set in concrete).

Unfortunately, as regards C++ my time has been diverted to other IT stuff, the Wordpress converter and very soon Purebasic development which will in time will replace Realbasic as the development tool for the Wordpress (and phpBB) converter. I'm getting a heap of hits relating to the C++ stuff and I'd hate to see it dry up.

Contributors do not have to be fully fledged C++ coding experts but proficient enough to know what they are talking about which translates to explaining why things are/were done in a certain way. Ideally in normal English, non sms speak.

What can I offer contributors? That's a good question, nothing but a sense of satisfaction.

I'm Australian based (Melbourne) and the server is in the US. So any local contributors would be very welcome but that's not to say international ones wouldn't be.

Okay, that's my pitch.

Steve
http://prime357.org and Looking for Contributors.

ps. Where I've mentioned C++, if you wanted, you could substitute PHP and/or Drupal itself.

Postgresql

Base 36 (Vancode) - leading digit/character

superjacent — Fri, 06 Jun 2008 00:13:04 +0000

I hope this is the right area for this question which refers to the 'thread' field of the table 'comments'. I provide a non php solution to convert Wordpress data to Drupal. I recently became aware that the thread field is not a string of decimal numbers but is in fact a string of base 36 (Vancode) numbers. I've created the necessary functions to convert to and from base 36 format but I'm a little confused re - the leading digit/character.

Here is an excerpt from http://api.drupal.org/api/function/int2vancode/5

Generate vancode.

Consists of a leading character indicating length, followed by N digits with a numerical value in base 36. Vancodes can be sorted as strings without messing up numerical order.

It goes: 00, 01, 02, ..., 0y, 0z, 110, 111, ... , 1zy, 1zz, 2100, 2101, ..., 2zzy, 2zzz, 31000, 31001, ...

So, decimal 35 = z and so it is written as 0z. The leading zero in this case, how does that indicate length? Therefore decimal 36 = 10 and so is written as 110 as per the excerpt above. Again, how does the leading character 1 indicate length when the length of 10 is 2.

Obviously I can see a pattern forming whereby the leading character for the first 36 is 0, the next 36 is 1, the next 2 and so on up until z. Is there an assumption that the leading character will never be greater than "z".

Any help appreciated.

Steve
http://prime357.org

Update: Since no response from the initial group (Database Schema API) I'm posting to other groups I think might be relevant. Secondary question is, for specific backend database type questions which group or forum is most relevant. I've got other backend database type questions waiting in the wings.

Postgresql

South Bay: Birds of a Feather session at MySQL Con followed by Sun & MySQL party (Both free attendance)

sooz — Mon, 14 Apr 2008 21:44:28 +0000

Join Amazon (Kieran Lal) and other Drupal community members and leaders at the Drupal Birds of a Feather session at the MySQL Conference & Expo on Wednesday, April 16th at 7:30pm. The BoF takes place in Ballroom “E” at the Santa Clara Convention Center. Afterwards, Sun will be giving away a Playstation 3 and Sun Fire X2100 M2 Server ($2,495) at their party next door. Wear your Drupal shirt!

More info about the conference:
http://en.oreilly.com/mysql2008

More info about the Sun party:
http://en.oreilly.com/mysql2008/public/schedule/detail/3338

Address:
5001 Great America Pkwy, Santa Clara, CA

Driving Directions and Map:
http://maps.google.com/maps?f=q&hl=en&geocode=&q=santa+clara+convention+...

Public Transportation:

Amtrak
There is an Amtrak station in Santa Clara at Tasman Ave. and Lafayette Street, underneath the Tasman Avenue overpass. The Great America Santa Clara stop is on the Capitol Corridor route with service to and from Sacramento. There are three trains daily. 1 (800) USA-RAIL.

BART
You can take the Santa Clara County Transit to and from the Fremont BART station. (510) 441-2278.

The Great America VTA Light Rail (www.vta.org) stop is directly in front of the hotel and convention center, and provides an easy connection to the San Jose International Airport, Downtown San Jose, Mountain View, and Caltrain. Using Caltrain you can easily connect to San Francisco International Airport, Downtown San Francisco, and BART.

High performance

Some review and proposal about Drupal 7.x database stack

hswong3i — Fri, 15 Feb 2008 06:40:02 +0000

I start my Drupal + Oracle research since the end of Drupal 4.7 life cycle (around Oct 2006), based on my client's request. The project is still running, but I am not satisfy about its progress. After keep trace in Drupal database stack implementation for more than years, it is time for focusing on Drupal 7.x development.

So what are the changes since 4.7.x? And what will be happened for 7.x? I would like to share my research progress with you, and so let's brain storming for what's next :-)

Drupal 4.7.x/5.x

Fig 1.1 Drupal 5.x database stack.

The case of Drupal 4.7.x and 5.x is very similar: we support both MySQL and PostgreSQL backend, with specific database access API file (e.g. database.mysqli.inc), enclosed with database.inc, and that's all. We still have some extra abstraction build on top of this simple model, e.g. tablesort.inc and pager.inc, for some useful application layer abstraction.

During this period, core and contribute developers need to understand the differences between MySQL and PostgreSQL, since database schema must handle manually. I would like to call this as the dark age of our history: we are asking every contribute developers as both MySQL and PostgreSQL expert (but most of us coming with MySQL-only skill set), or else code was not portable. This result in a complicated PostgreSQL support, and so most contribute modules are MySQL only.

Drupal 6.x

Fig 2.1 Drupal 6.x database stack.

For our newest stable Drupal release - 6.x, Schema API is being introduced. We spend a lot of time for its development, during 6.x 1-year development life cycle. It is target to solve the schema problem as mentioned above. Contribute developers are no longer required as bi-expert nor multi-expert: we just need to know how to define our schema in a correct Schema API syntax, Schema API will handle the rest for us. IMHO, it is very powerful, and it is an important improvement for other database development.

drupal_write_record() is being introduce, too. This handy function will grep out existing cached schema, and build INSERT/UPDATE queries for us automatically. It try to handle all column type binding for normal developers, which is usually very bulky in implementation.

BTW, the work is not too elegant: all stuff are mixed into database.*.inc, and so files are even split as 2 section (groups database and schemaapi) internally. This may increase the difficulty for our core maintenance.

Most likely, this design is very similar as Moodle 1.7+ XMLDB implementation.

So what's coming next for Drupal 7.x?

Fig 3.1 Well... So what's coming next for Drupal 7.x?

The truth is: I don't know the overall schedule up to this moment. Data Architecture Design Sprint group is now hard working for this, and going to present their research progress in Boston DrupalCon 2008. Since we are going to use PHP PDO (instead of legacy PHP database connection drivers), people are also focusing on implement Data API or Active record model for a higher level of abstraction. Before they have their final decision, I would like to provide some successful case study as reference, and let's brain storming for the optimal solution :-)

Fig 3.2 Moodle 1.7+ XMLDB stack. From doc.moodle.org.

When we are talking about improve our database abstraction as more powerful and universal, Moodle 1.7+ XMLDB stack should act as a perfect example (it since Nov 2006). As Moodle core uses ADOdb internally, this possibility has been present since the beginning. Plus their XMLDB abstraction, Moodle 1.7+ is able to handle both MySQL, PostgreSQL, Oracle and MSSQL while maintaining everything working properly. I will not detail their implementation within this article, and you may refer to their introducetion, roadmap and problems document.

The most important idea of Moodle's XMLDB stack is split-brain handling. They left every layer keep focus on their duty ONLY, and so the final approach is perfect and elegant. This is also similar as the well-known OSI model: A layer is a collection of related functions that provides services to the layer above it and receives service from the layer below it.

Fig 3.3 Research progress of Siren + personal suggestion.

Well, Siren is my personal research project besides Drupal 6.x, which target to explore the possibility of multiple database support as like as Moodle, Gallery2, and eGroupWare. Moreover, prepare research progress whenever Drupal 7.x is open for public development (and that's now). Within this project, we are able to support both 5 database + totally 9 PHP database access drivers, including mysql, mysqli, pgsql, oci8, pdo_mysql, pdo_pgsql, pdo_oci, pdo_sqlite and pdo_ibm.

It sounds interesting, isn't it? So what changes I have been made in order to support this approach? Just some step forward improvement: split the handle of function calls into number of individual files, and each layer just keep focus on what they are needed for. E.g. split Schema API related functions into schema..inc, common DML into common..inc, and so on. Surly I plug some extra update to our core queries syntax; but most likely, the idea is just very strict forward. The research progress should belongs to OSI session layer when compare with OSI model.

IMHO, Data API is a must for our future, if Drupal is going to be a web service framework. Acting as a high level abstraction should be the best position for it. It should be similar as the idea of OSI presentation layer, which is one level below our core application APIs. It may implement as OOP or whatever; BTW, according to its duty and responsibility, it should be separated from PDO database access API (where PDO is much similar as OSI transport layer implementation).

One statement as conclusion: split-brain handling.

Any else suggestion?

Database driver is something only need to be implement once and forever. Once the design of logic is completed, the rest is all about the style of implementation, e.g. using OOP as ADOdb or traditional function callback as Drupal.

The design of abstraction layer is also similar, too. Once you get a complete study of the variation between each databases, and able to foresee ALL of the needs of their specific limitation, it is all done. There is no such idea of "Contribute Supported Database" within the above 2 layers, and it is what I call as "No gray area, but only black and white". Whatever you support it, or else you don't.

On the other hand, Data API is much different. It try to serve the need of our application, and so its requirement will keep on changing depend on time. Therefore a split-brain handle should be the most suitable implementation. Please correct me if you have some better idea :-)

But the truth case is: we are lack of PostgreSQL/Oracle/DB2/SQLite/etc developers, and not enough task force nor union for a complete and perfect support for them. Up to this point, I would like to proposal the idea of "Contribute Supported Database" in this way:

Implement ALL possible database drivers into core, and build an abstraction that cover ALL the needs of them. This is a once forever task, and the base of the rest of discussion.
ALL core libraries and essential modules for installation MUST implemented as universal, so basic installation MUST function for ALL databases. E.g. node, comment, users, block, etc. This is all the duty of our core developers which need to be ensure before stable release. On the other hand, if MySQL (and maybe SQLite, too) is our main focus, just keep its support as perfect as usual.
P.S. an additional pre-requirement for this point: try to implement ALL core module as SQL-99 compatible by default, and so save a lot of time for further more follow up handling.
Label the optional modules as "Not supported" or "Not perfectly supported" within modules selection page, if that is the case of specific database backend; give a different level of support status; or maybe even, disable its activation checkbox by default. Don't let our second/third priority database hold our stable release!
left the issue open for contribute developers if they really hope to let that module function for database A/B/C/etc, and release subversions besides stable release for this purpose. E.g. if now search and forum module are not compatible with Oracle, just let it be as Oracle developers duty!

This is my ideal proposal. So how about yours? Here is the raw file of above images, please feel free to utilize it if you hope to visualize your idea as above :-)

Postgresql

DB support as contribute: is it a good idea?

hswong3i — Wed, 13 Feb 2008 08:45:44 +0000

Abstract

Drupal 6.0 is revamped with Schema API, so what's next for Drupal 7.x? PDO for sure! With this powerful data-access abstraction layer, workload will much reduced for DB abstraction layer designers and developers, and finally benefit our contribute developers and end users.

By the way, together with the decision of Drupal 7.x + PDO, there is also some voice about moving PostgreSQL (and so other potential databases support, e.g. Oracle, DB2, MSSQL, etc) support away from core, but contribute; on the other hand, add official SQLite support into Drupal core, together with MySQL.

Is this really a good idea? Or even if it is possible? As an existing Drupal + PostgreSQL users, what will this affect your daily work? As a potential customer of Drupal + Oracle/DB2/MSSQL/etc, is this a good new for you, or just an evil? I would like to provide some brief idea for you within this article.

Background

After more than 6 years of PostgreSQL supporting in Drupal core, PostgreSQL users are going to have a revamped experience with Drupal 6.0. Drupal 6.0 will ship with Schema API, which simplify most of the variation between databases handling - DDL. This useful improvement will not only benefit our existing core and contribute developers, but also simplify most workload for other database driver development, besides MySQL and PostgreSQL. With Drupal 6.0, we can simply foresee the expand of existing market sharing, together with the explore of hidden customers.

As Drupal 6.0 is ready to launch, it is also a good timing to plan for Drupal 7.x. As we will only support PHP 5.2.x or above in Drupal 7.x, it is for sure that we will revamp our existing DB abstraction layer with the help of PDO. It does simplify and standardize a lot of function call implementation, together with some useful new features. If you are a DB abstraction layer designer and developer, moving your work to PDO will save you a lot of time.

On the other hand, some people also think it is a good timing for adding Oracle, SQLite, DB2 and MSSQL support into Drupal core. Most of this request are halted for 2~3 years.

Some of the developers are hard working with the future plan for how Drupal 7.x DB handling should looks like. BTW, the idea is most likely for expand Drupal usability and functionality, but not about maintain nor expand different databases supporting: some developers even think we should only official support MySQL and SQLite with Drupal 7.x + PDO, but move away PostgreSQL (nor other potential databases) supporting as contribute level.

What PDO Does and Doesn't?

For finding out why this brain storming happen, we need to have some idea about what PDO does and doesn't:

PDO is a data-access abstraction layer, but not provide a database abstraction. E.g., this means we can simply forget the function API differences between mysql_connect(), mysqli_connect() and also pg_connect(): we just need to construct a new-standardized PDO object with different DSN (Data Source Name). Moreover, we can forget those complicated programming logic differences between pg_escape_bytea(), OCI-Lob->load() and db2_lob_read(), but simply using standard Stream API as PDO LOB (Large OBject) handling.
BTW, these are most likely abstracted by Drupal 6.x DB API, e.g. db_connect(), db_query(), db_decode_blob(), etc.
PDO provide prepared statements and variable binding, even it is not supported by database, e.g. MySQL. With PDO variable binding, we may use ? or named-variable (leading with :, e.g. :value) within prepare statement. This seems to be a good idea for replace our existing user-space printf-syntax (e.g. %d, %f, %s, %b and %%) as PDO's C/C++ implementation, and simply forget the relationship between data, data type and database column type. This also give a great hope to Drupal 7.x Data API research and development.
So isn't it perfect? No, it is not... First of all, most databases require special LOB handle during variable binding (specific PDO::PARAM_LOB manually), e.g. PostgreSQL, Oracle, DB2, etc. Only MySQL and SQLite are able to handle variable binding without type specified. Also, there is no spotlight performance different, even we swap queries variable binding from user-space printf-syntax to PDO, which is proved by benchmarking result.
PDO isn't a database abstraction layer, it doesn't abstract and standardize ANY database variation. E.g., PDO doesn't solve the problem of reserved words conflict, it don't abstract any SQL function name variation, it also have not duty with resolve the maximum string size limitation across most databases.
Let's take Oracle LOB handling as an example: PDO give none of help to Oracle LOB INSERT/UPDATE handling, we still need to use RETURNING + Schema API. According to my research progress and case of Oracle, using standardized function call is the only benefit of using PDO, but didn't help or simplify any programming logic or implementation.

IMHO, PDO is very useful for DB abstraction layer designer and developer: it do simplify most of the DB function call variation; BTW, when we are talking about "PDO can improve database abstraction and so software with PDO can support more databases", it is not really truth. PDO don't have duty with this ;-(

How about Drupal + PostgreSQL (or other databases) support as contribute?

It is for sure that MySQL own around 90-95% of our existing market sharing; it is not question that SQLite is a manifest typing database engine; based on the above PDO stuff information, MySQL and SQLite are both very suitable with PDO lossy-form variable binding technology (able to use ? and named-variable binding without type specified). But when we are talking about "How many percentage of PDO technologies are utilized by MySQL and SQLite?" and "How many percentage of technologies that PDO NOT covered are used by MySQL and SQLite?", I guess the answer may just below 70%...

So what will happen if we seem MySQL and SQLite as our first priority core supported database, and ONLY focus with them? For sure that we will miss out the rest of 30%, which means the basic elements needed by other databases, within Drupal database abstraction layer. This also means asking other database drivers implement as contribute is just a nonsense, if we only support a subset of requirement within core.

On the other hand, writing MySQL-specific queries also generate a lot of incompatible issues for PostgreSQL during Drupal 6.x development life cycle. People also claim that PostgreSQL issues slow down our Drupal 6.x release. BTW, there is still a lot of incompatible issues even PostgreSQL is now seems as our "Official Supported Database" in Drupal 6.x, we can simply foresee about the case if we just seems it as contribute supported. It is better say that: we are NOT going to support PostgreSQL anymore! And so the similar case for other databases driver development: we will have NO WAY to implement other database drivers as contribute support!

PDO + PostgreSQL + LOB handling should be a good example of this assumption. In case of pdo_pgsql, we need to specify PDO::PARAM_LOB manually during INSERT/UPDATE variable binding, and must use Stream API for BLOB decode. Both of these handling are NOT required by MySQL and SQLite. If our Drupal 7.x Data API design are target for MySQL and SQLite ONLY, it is for sure that NO WAY for PostgreSQL driver implement as contribute: we will have no hook for it!

As a simple conclusion, multiple database support MUST build inside core, with a complete research of what are they needed for. There is no shortcut, and no gray area. The support of database is just simply a TRUE/FALSE question. Please don't be faked by the beautiful wording of "Contribute Supported Database": it is just a sweet poison candy, that is not the truth case, and I have NEVER seen a successful story about this :-p

Oh my god! So what should I do for it?

If you are an existing Drupal + PostgreSQL user, it is strongly suggest to keep you eye focus on the Data Architecture Design Sprint group, comment your needs and wish to them, or else you may be fired away without notice. You may also have a look about issues related to PostgreSQL, give a hand to it if possible, and comment them as a Wiki collection. IMHO, it is a critical timing for Drupal + PostgreSQL besides the past 6 years support period. It is time for YOU to take YOUR action, or else will be too late.

If you are a potential customer or user of Drupal + Oracle/SQLite/DB2/MSSQL/etc, beside the above suggestion for PostgreSQL, I would also like to invite you to give a hand on some existing issues, review and comment them, and finally record your work in another Wiki tasklist. According to some research founding, we would able to enrich our database supporting without a critical pain of revamp, based on our existing Drupal 6.x DB API implementation. Most research are now completed, so issues review and comment your point of view should be your highest priority. Please remember that: comment is power, and YOUR review is very important for speeding up the core support of your target databases.

My humble conclusion

Every developers have their own background, their own wish, their own needs, and also their own rose garden. I would not like to comment this "PostgreSQL as Contribute" idea as silly; but at least, it is not my cup of tea.

On the other hand, supporting more database won't be a conflict with the research and develop of Drupal's Schema API or Data API: they are target for a better developer and user experience, serving database abstraction should be one of their core duties. Asking for better abstraction and functionality but reduce the number of supported database, or even close the door for other database should be count as loss focus.

From my point of view, more choice is always better than just giving me a single solution. The use and choose of database should depend on the needed of each individual case of clients and users, but not forced by developers. If we are already close to our target, there is no point for rollback our 6-years progress on tomorrow.

A beautiful-sweet-poisoned candy? No thanks!

Postgresql

Issues Need Review for Enhance Cross Database Compatibility

hswong3i — Tue, 05 Feb 2008 08:48:16 +0000

Enhance cross database compatibility is important for our Drupal Core, especially when we are going to ship D7 with PHP5.2.x PDO implementation. BTW, this will require number of individual patches, so your help and review are required - add it to the top list with a brief description and an estimate of the time to review. Once reviewed, move the issues to the bottom "reviewed" list, perhaps with your name attached.

Need Review

http://drupal.org/node/371 - Resolve ANSI SQL-92/99/2003 reserved words conflict. (just a few minute, but need a lot of follow up)
~~http://drupal.org/node/199217 - Split Schema API and shared functions from DB drivers. (just a few minute)~~ (Already come with TNG DB)
http://drupal.org/node/167335 - Handle database-specific SQL functions/operators by simple abstraction. (just a few minute)
~~http://drupal.org/node/199101 - Core queries coding style cleanup for PDO development (should be RTBC, but need a lot of follow up)~~ (Already come with TNG DB)
~~http://drupal.org/node/218128 - Enhance _db_query_callback() for PDO.~~
~~http://drupal.org/node/183148 - INSERT/UPDATE/DELETE API, drupal_drop_record() and enhance drupal_write_record().~~ (Already come with TNG DB)
http://drupal.org/node/147947 - Replace some TEXT:BIG with BLOB, based on cross database compatibility concern.

Reviewed

Committed without Thorough Review

Postgresql

My personal battle target for Druapl 7.x

hswong3i — Fri, 01 Feb 2008 03:45:51 +0000

My primary battle plan should be enhance Drupal cross database compatibility. I have involved in this topic for around a year, and I would like to keep it on going. To accomplish this target, I would like to complete the following tasks before D7 code freeze:

ul>

Improve Database API for better abstraction

Something should be done before any other targets, e.g. resolve reserved words conflict, split drivers into individual files for better management, indeed but simple abstraction for different database syntax, add INSERT/UPDATE/DELETE abstraction and enhance drupal_write_record() abstraction, etc. With a better common base for multiple database development, we will need much less time to study the differences between databases than that of before.

Add PHP PDO supporting

Shipping D7 with PHP PDO should be a spotlight topic, since it is now official package for PHP5.2.x, and legacy drivers will soon be removed in PHP6.x. We may need some cleanup in our core query syntax, in order to let PDO get works.

Introduce more database backend

Up to this point, I would like to introduce some other database backend for D7, e.g. Oracle, SQLite, IBM DB2 and also MSSQL. If we are able to support SQLite in D7, which also means we are going to standardize our core queries into SQL92 standard, the problem for maintain multiple database backend should no longer a critical problem; if we are able to support Oracle, the most complicated database I guess, this should no longer a problem for us to implement drivers for other databases.

I have summarize most of my personal battle targets in here. Most logic are proved as functioning, and they are all get set for the open of D7 public development. On the other hand, I would like to explore if this may integrate with other interest musings with Data API, so we will able get all stuff better in D7 ;-)

Database

Are performance optimisations still going into D6?

robertDouglass — Mon, 28 Jan 2008 15:07:59 +0000

I'm assuming it's too late, but if not here's an easy performance win for D6:

http://drupal.org/node/215080

The type column of the {system} table is currently a varchar(255) field. Yet in 99% of all Drupal installations it stores either the text 'module' or the text 'theme' and nothing else. This is already a strong case for making it an int column and defining constants for module and theme. Then, on every page, there is this query:
SELECT filename FROM system WHERE name = 'user' AND type = 'module';

This query always takes over 1 ms to run making it not a criminally slow query, but relatively slow, nonetheless. This is because the list of indexes available doesn't help the query much

By shortening the {system}.type column to 32 chars and adding an index we reduce the query by around 60%... and this query runs on every page load.

Postgresql

Musings on a Data API

Crell@drupal.org — Sun, 06 Jan 2008 03:38:22 +0000

I have been pondering the question of a data API for a while, as have a lot of people. Much of the recent discussion has focused on an Active Record approach to a data API. Now, Active Record is a very powerful architectural pattern. It maps nicely from storage to interface, it can be fairly self-documenting, and it is conceptually simple and approachable.

It is also, I believe, insufficient.

Active Record

The basic concept behind Active Record is a one to one mapping from a database record to a user-space object. That object can then be manipulated as any other object, and then saved back to the database with all SELECT, INSERT, and UPDATE statements built dynamically. Additional features, like automatic foreign key lookup or multi-load capability, are also possible, particularly in PHP 5.

Active Record, however, has a number of limitations. First is the raw one to one mapping of database fields to object fields. Very often, you want to work with data at a higher level than its database representation. That can be partially solved by adding additional utility methods to a class representing a given table or entity, but that still assumes that there is a close binding between a table and a first class entity in our data model. Take a typical Drupal node structure. While every node has a record in the node table, it frequently has multiple. Nearly every node also joins against other tables depending on the value of the type column. That's not information we can know before we load the record in the first place.

The second issue comes with multi-value fields. Multi-value fields, in SQL, are typically handled with a dependent table keyed by the primary table's primary key and either a local surrogate key or the value itself. Have a look at CCK's dependent tables for a textbook example. They are keyed by nid, vid, and a local delta value.

Classic Active Record doesn't handle multi-value fields directly well. Primarily, it is not possible to load an entire object/record with its dependent fields in one query efficiently. The primary table should return a single record, while the dependent tables could return any number of records. While it is possible to run a join on the two tables, that results in a lot of duplicate data being retrieved on each load; the primary table's fields repeat to match each entry in the dependent table. Now add in multiple multi-value fields and it becomes painfully clear that a single-query load operation for any non-trivial data structure is not possible.

That leads us to a very important conclusion: Loading a rich data object from the database will always require multiple queries per load, whether they're loaded all at once or via a lazy-loading mechanism to pull in dependent fields only when needed. Lazy loading also requires a layer of abstraction behind which the lazy loading may occur.

The third problem of Active Record is that it presumes the data lives in a local SQL database. It is, by definition, a relational-database-centric design. Data, however, is not always database-centric. Files rarely live in a relational database, and practically never in Drupal. As I write this I am working on my second project recently where the bulk of the data that I wanted exposed as a node does not live in Drupal but in a 3rd party system. In one case it was a foreign (non-Drupal) system with its own API and database connection. In the other, the data is coming in from a SOAP connection. Active Record would not have supported either scenario.

Now, that's not to say that Active Record is a bad design pattern. It is quite useful in a variety of cases, and I've written a number of implementations of it myself (to varying degrees of quality). Drupal 6 now contains the beginnings of an Active Record-style utility suite in the form of drupal_write_record(). However, for a system with the flexibility and potential of Drupal it is too limiting.

Object-Relational Mapping

If Active Record is a database-centric view of data storage, an Object-Relational Mapper, or ORM, is a user object-centric view of the world. In an ORM, the storage of an object is independent of its consumer-facing structure and interface. If it happens to map 1:1 to a database record, that is coincidental. That gives it a lot more flexibility, but also a lot more complexity.

Because an ORM is more robust, it can do a lot more optimization under the hood. Values can be moved around between tables (as CCK does) without changing its interface. Complex relationships can be modeled in a database-independent way, even tying into non-SQL systems like the file system, SOAP, etc.

The flip side, however, is that an ORM can be quite complicated for complex queries, even more so than an Active Record setup. Finding all "event nodes created by Bob in March that take place between May and July and have at least 5 attendees" is reasonably straightforward in SQL (albeit a bit verbose). In Active Record it is more difficult, but can still be done if you can access the SQL layer underneath it. In an ORM, such a query becomes very... involved. Some ORMs, like Doctrine, develop their own query language to address that problem, which in turn means parsing and interpreting a query language in user-space. PHP is not known for its string parsing and interpreting-fu. That's why SQL databases are so popular; they are.

Drupal as of version 6, however, does implement an ORM. Nodes are not built Active Record style. Hooks can add data to nodes from anywhere they want, whether it maps to a database table or not (although it frequently does). However, the ORM is "naked". That is, there's no abstraction on data access. Everything is loaded at once into a single blob-o-raw-data that gets returned from node_load(), and then it's up to modules to not bump into each other and module authors to know what to do with that raw data.

Nowhere is that more painful, in my experience, than in creating your own node form. Drupal saves the submitted form data back to the node object based on the form structure, not the node structure. Matching those up is entirely the module author's responsibility, despite the fact that the form structure and node structure frequently have no right to be the same. If they don't match up, you end up with a hybrid $form_values and $node object. That's just weird.

Thingies

Despite their additional complexity, I believe an ORM approach to be better long-term for Drupal than an Active Record approach. However, that does not preclude a hybrid approach.

What would a hybrid approach be? A hybrid approach would involve a Thingie object (that was the working name decided on in Barcelona for first-class entities like nodes, users, files, etc.) that defines multiple fields. Every Thingie is restricted to a list of fields and properties. Each field can lazy-load itself on demand. Each field is also its own object, and can either be implemented Active Record style (tightly bound to the database with lots of automation) or not (to support SOAP or XML-RPC or flat file or even calculate-on-demand fields). That is, everything becomes a self-containing CCK field.

For nodes, that would mean non-CCK fields largely cease to exist. Each field, however, would have much more flexibility and could potentially return a processed value or array. For users, the profile module as we know it gets replaced by the same structure as nodes would have, potentially even with the same field objects. Files would finally become first-class Thingies, although would be unlikely to have very complex fields. (But then, who knows?)

The ability to return complex values is important. Some modules add fields conditionally. Ideally, they should do so under a module namespaced array. That is, $node->mymodule['some_field']. With an "all CCK" model, $node->mymodule becomes an object of its own which could have its own single ->value, multiple properties, even its own methods if necessary. Some standard methods (which are more flexible than properties) would be a good idea as well for standardized access, especially for theming.

For multi-value fields, there are two options. One, all fields could be multi-value. That's how CCK works now. CCK fields have a structure of $node->field_foo[x], where x is a numeric delta. After the node has been rendered, each delta has a ['view'] property that has the rendered value of the field and then some combination of other properties for the raw data field(s). That makes handling multiple values more straightforward but can be annoying to work with when most of your fields are single-value. Two, multiple fields could be listed as separate fields. That is, if a node has a single Foo field and two Bar fields then they would be accessed as $node->foo_0, $node->bar_0, $node->bar_1 rather than as $node->bar[0] and $node->bar[1]. That would make treating each field instance separately easier, but referencing an set of fields together more difficult.

I believe CCK got this one right. The flexibility and consistency of treating all fields as multi-value is worth the more complex syntax. However, the syntax could potentially be simplified. PHP 5's Iterator interfaces could be useful here.

Objects

So far I have assumed that we are talking about a system built on classic objects (that is, objects of a class other than stdClass), not on giant arrays. Certainly, everything discussed here could be implemented using no classes at all. Any problem in programming can be solved either by procedural or OOP programming (or functional, if PHP supported that). However, certain problems are far better suited to one style or the other, where "better" could mean any of less code, more performance, easier to understand, more secure, more flexible, etc. A complex data model, such as the one Drupal now supports, is, I believe, a case where Object-Oriented principles using well-factored classes and methods is a "better" fit. As I've said before, the right tool for the right job. In this case, objects are the right tool.

That does not preclude function wrappers where a function would be more convenient. For example:

<?php
function node_load($nid) {
  return Node::load($nid);
}
?>

Until now there have been very good reasons to not implement heavy OO in Drupal; primarily, PHP 4's object model didn't offer any compelling reason to, as it was too primitive and limiting for the sort of functionality we would need out of it. That is no longer the case in PHP 5.2.

The other major factor is that Drupal's current architects come from procedural backgrounds, so Drupal is mostly procedural, so it attracts procedural thinkers, etc. That is not inherently bad, of course, but at the same time we should not allow that legacy style to hold Drupal back any more than we allow legacy code to hold us back.

Another advantage of a good (emphasis on good) unified Thingie system is that it would make it easier to create new Thingie types. Sometimes a one-off project could benefit from a readily-available data API that is not nodes.

Variants

In Barcelona, Jeff Eaton and I spent a fair bit of time bouncing around concepts for an OO Data API. The main idea that we struck upon was the concept of "variants". A "variant" was an alternate version of a given Thingie. For instance, Revisions of a node are variants of it. For a file, at a trivial level different resolutions of an image are different variants. However, that could easily be generalized. Picture a single File Thingie that is a YouTube-esque video. One variant of it is a high-res version. Another variant is a low-res version. Then there are small, medium, and large thumbnails of it in JPG format. Then there's an English, Spanish, and Chinese transcript, as PDFs. And so on.

For nodes, revisions and translations would be the obvious variants. Variants would also need to stack. That is, one could have node 12, revision 2 of the English version and revision 4 of the German version, but all with the same node id. That would eliminate one of the largest annoyances of dealing with content translation currently: keeping track of nodes that are translations of each other.

Each variant would have to have a default, especially since ideally we want variants to be defineable by modules rather than hard-coded. So for instance, one could load a node like:

<?php
Node::load(12, array('lang' => 'es', 'revision' => 2));
?>

That is, load node with id 12, the Spanish variant, the revision 2 variant, and default for all other variants. If lang were not specified, it would mean the English variant. If revision were not specified, it would mean whatever the most recent revision is.

However, that quickly runs into a problem of storage. While a nice API concept, it makes the SQL model more difficult because you then have potentially variable primary keys. That gets ugly, fast.

Perhaps those should be hard-coded...

Rendering

Although one always wants to keep data logic and display logic separated, rendering a Thingie for display is perhaps its most common use. That means we need to consider how such an API would be used for rendering. The natural knee-jerk reaction would be something like this:

<?php
$node->render();
?>

It would also be very wrong. For one thing, it's very brittle. What happens when you want to render as a full node vs. teaser? As an RSS feed item? As a JSON response? Render different node types differently? Very quickly you end up with a whole bunch of flags on the render() method, and we're right back where we started.

More fundamentally, however, it is pulling rendering into the data object. That means there is poor separation between data access and the presentation layer.

Rather, we should take advantage of the fact that, in PHP 5, objects are cheap and passing objects is cheap. The data access object itself should remain pristine, but can then be passed around to other functions and objects at nearly no cost. We can also take advantage of the strict, rigid structure of the Thingie object as a collection of fields. That is, we don't need to render the Thingie! We render its fields and aggregate them for the theming layer.

<?php
function theme_node($node, $style = 'html', $mode = 'full') {
  foreach ($node->fields() as $field) {
    $return[$field->name()] = theme('field_'. $field->type(), $field, $style, $mode);
  }
  return $return;
}
?>

Grossly over-simplified, but you get the idea. ($style would be things like "print", "html", or "rss" while $mode would be "teaser", "full", etc.)

Ready, Set, Go

One of the big drawbacks of an ORM, particularly one built on lazy-loading is the risk of the "SELECT 1+N Problem". That is, imagine the following code:

<?php
$books = find_some_books();
foreach ($books as $book) {
  print $book->title .', '. $book->getAuthor()->name();
}
?>

Assuming that Author and Book are two separate entity types, we will execute one query to find the books we want (with its title), and then for each book we will execute another query to load the corresponding author object with the author's name. That is a performance killer. Generally, there are two alternative approaches:

Greedy-Loading: The process of loading a Book or Books will also load all related data, period, with whatever joins or additional queries are needed. That means it can write more efficient queries, but it's also loading far more data than it probably needs. Do we really need to also load all of the Publisher information? Doubtful. This is actually what Drupal does currently, although it doesn't even optimize multi-object loads (unless you're using a List View, which then acts as a very optimized but complex ActiveRecord-ish loader, not an ORM).
Semi-lazy loading: If you know in advance that you're going to want the Author object as well, you can specify that the ORM loader should also pull that data. That is, you write a JOIN, but you don't use SQL but some ORM-specific directive, usually a very verbose one. As I see it, that loses much (though admittedly not all) of the advantages of a lazy-loading ORM: Don't make me think. Then if you need to pre-load against multiple other fields, well, pretty soon you have code so complex that you may as well just write your own SQL and be done with it.

And neither handle multi-value fields, of course.

I would propose a third, more flexible approach; let's take a cue from both jQuery and CCK. A jQuery object (the $() thing) is never a single entity. It is always a set of DOM nodes, sometimes a set of one DOM nodes. In CCK, a single-value field is, structurally, the same as a multi-value field that only happens to have one value. Views are all about multi-object queries, although those can also frequently have only one entry. They're still treated as multiple entries.

Set operations are both extremely powerful and very natural. SQL itself is built on set operations. Many RPC requests either assume or allow set operations. There is, in fact, no reason why a single-entity operation cannot be implemented as a set operation.

OK, not entirely true. If you have a naked data structure, then you never have an entity but an array of entities, which you then need to foreach(); that can be clumsy if you don't practically use multi-value fields often. That's another place where having an OO interface is "better" (in this case easier) than a procedural one. You can encapsulate the loop, much the way jQuery plugins do. That makes the module-developer-facing API cleaner. This is a good use-case for Interfaces:

<?php
interface Something {
  function doSomething();
}

class Thingie implements Something {
  function doSomething() { ... }
}

class ThingieSet implements Something {
  protected $thingies;
  function doSomething() {
    foreach ($thingies as $thingie) {
      $thingie->doSomething();
    }
  }
}

$t = new Thingie();
$t->doSomething();
$s = new ThingieSet();
$s->doSomething(); // Does something to every thingie, even if there's just one of them.
?>

But why do we even need a loop? Moving a loop inside a function call makes the API cleaner, but especially when dealing with SQL databases doesn't necessarily help performance. Remember, though, that we're demanding a rigid Thingie structure. A Thingie is composed of Fields. Let those Fields handle their own set operations, especially for lazy loading. That means each field needs to know what its "load set" is. That is, suppose we have a Set of Node objects. When we access, say, the taxonomy field on one node, it should lazy-load all taxonomy data for the nodes in that Set in a single query and then assign out the data to the appropriate node's field. Update operations then can also be grouped, making it possible to load a set of Thingies, set one field on all of them, and then save, with the load and update operations both taking only one query each no matter how many Thingies we're modifying, and without writing any SQL ourselves.

I've implemented such a system recently for a 3rd-party integration project, and it is actually not difficult to implement. (Well, the read part, anyway. The write part is more complicated, but I believe it is solvable.)

While not as efficient as explicitly specifying the fields we want, it does flatten the cost. That is, the Book and Author example above would have a cost of 2: one to mass-load books, and one to mass-load authors. The worst-case scenario for such an approach is that every possible field gets called anyway, or 1 + F. That is exactly the cost of a node_load() in Drupal now... except that node_load() is actually N*(1+F) because it doesn't do set operations.

Searching

The other major drawback of an ORM is that non-trivial searching is frequently very verbose. We need to specify a variety of weird search parameters in a completely data-agnostic fashion. That means lots of methods, or lots of objects passed into a long method signature with lots of arguments, such as in Qcodo. Or it means a custom query language like Doctrine. Both are icky.

To be honest, I see the best course of action here to be "punt". Separate loading from searching entirely. A set-load operation should take simply an array of IDs. How those IDs are found is a separate question. They could be from a simple API call, or looked up from a field in another object, or from a Views-like builder, or from a direct SQL query builder, or a hand-rolled 15-line super-complex SQL query (which would be about 1/10th the size of trying to write that query abstractly with fully-OO directives), or retrieved from a remote server via SOAP. By keeping searching and loading separate, we maximize flexibility while minimizing complexity. While there will be, in some cases, a cost in terms of additional queries, that cost is relatively flat and, I would argue, a good value.

Now, combine flexible searching with everything-as-a-set, and you have a very large portion of Views not just in core, but being core itself.

What not to do

There are, of course, a lot of potential pitfalls in any such endeavor. There are two I will mention in particular.

Deep inheritance. Deep inheritance hierarchies are well-understood as being dangerously brittle. Adding functionality requires editing existing code. You can't do things "sideways", as Drupal does nearly everything. If you want a certain piece of functionality to be part of two classes that otherwise have no reason to be logically children of each other, you're SOL. That's not to say that inheritance is always bad, but it is very easy to get carried away. Let's avoid that.
For example, it may seem natural to have a Node class, and then child classes of Node for Page, Story, Event, etc. However, that's not how nodes are actually used in Drupal. A node is a node is a node, but it has some combination of fields on it (even now). Rather, a Node should continue to have a ->type, which in turn indicates what field objects should be added to a generalized field container.
Code generation. I have never understood how the "Don't Repeat Yourself" and code generation design philosophies can coexist. Code generation (speaking of logic generation here, not function skeleton generation) is repeating yourself. In fact, its repeating yourself so much that you make the computer repeat yourself for you. That's a sure sign that you are doing something wrong, and need to refactor your code to actually use inheritance, or composition, or something other than typing out the same code over and over again. Of course, there is one catch here: PHP 5.2 and earlier does not support late static binding, which means, in essence, static methods can't be usefully inherited because references to other static methods will always be to the defining class, not the actual class. Fixing that is a hotly debated topic for PHP 5.3, but that's still well-off as far as we're concerned.
Tight-coupling. (Nobody expects the Spanish Inquisition!) A truly pristine data API will require fully and totally divorcing it from the rest of the system, particularly forms. The node edit form's save routines, for instance, should not in any way shape or form know anything about SQL. They should all simply set values on the $node object, and then eventually $node->save() gets called and fields internally save themselves. Similar separation is also necessary for theming. (Like, can we get comments out of node_view(), please?)

The future

What I have described here is, of course, ambitious. Very ambitious. It represents a fundamental shift in the way Drupal handles data. However, I believe that shift is necessary for Drupal's long-term success.

Dries has set some rather lofty goals recently. Web services. Drupal as the "Linux of the web". He doesn't think small, and neither should we. A robust, powerful, and above-all decoupled data API moves Drupal from being a CMS or application framework to being something else: A data server. The menu system acts as a simple router to handle RPC/REST requests, which the data server responds to. It just so happens that the most common REST request is for an HTML page; at least for now. In 5-10 years, though? Will we even be talking about SQL and HTML? A good data API should be agnostic of both.

Implementation

So what would such a system look like in practice? How would we use hooks with it? I'm glad you asked! I'm not entirely sure yet, aside from the samples above. I have some ideas, however, most of which owe thanks to other members of the Drupal community. For instance, courtesy of Jeff Eaton (who may credit it elsewhere himself) is hook_nodeapi() going away in favor of hook_node_info() defining what field types a given node type has, and then those field types (objects) internally do whatever loading and saving they need. Adding fields to a node type is accomplished by a simple hook_node_info_alter(). hook_user() would likely suffer a similar fate.

One of the possibilities discussed in Barcelona was to implement just files using a new Thingie-based system, as a trial balloon. Despite recent improvements, Drupal's file handling is still a weak point. That means the risks of experimentation are lower than if we were to break, say, nodes. If it works out, we can then extend it to the rest of the system in the next future, and we'll have a good API behind which to implement things like, say, pluggable storage engines.

At this point, I welcome feedback from all and sundry on the concepts presented here. Truthfully, I do not expect it to all happen in Drupal 7. If it all happens by Drupal 8, it will be aggressive. However, we do need to have a clear picture of where we are going and how to get there, even if it will take a few versions.

Database

MySQL Table Type

webappl — Tue, 01 Jan 2008 23:57:02 +0000

I think it is safe to say that many developers already using or about to start using MySQL table type InnoDB. Some developers even changing default MySQL storage engine to InnoDB.
In many cases it make sense to convert table type if not for all tables, then at least for some tables.
In this situation will be nice if modules developers will state what table type is necessary for those modules and why.
Otherwise, its has to be investigated and tested for all modules in the project installation. May be it is fun but it is also a lot of time.
My suggestions for Drupal developers are

include into modules .install file the indication of MySQL table type in the end of each table creation query like "CREATE TABLE `tablename` (...) ENGINE=InnoDB DEFAULT CHARSET=utf8" or "... ENGINE=MyISAM ...";
include into 'INSTALL.TXT' file description of why these tables need to have those types.

That will give another developers enough information for making right decision on MySQL optimization of project's Drupal installations.
Otherwise, it has to be investigated for each table.
Please share your experience.

Alex Pandre
Custom Drupal modules and themes development, SVN for Drupal environment development, Code optimization
Join MySQL group and post your suggestions in related groups in the same time

MySQL

PHP 5.2.5 breaks SQL Server support; Yet MSFT SQL Driver Works

Souvent22@drupal.org — Thu, 20 Dec 2007 18:32:32 +0000

So, I've had some free time latley, and I've started back on the database API layers (DB2 and SQL Server currently).
However, I have found 2 disturbing issues.

Overview
First, I'm currently exploring the following 3 ways to connect to sql server:
1. FreeTDS
2. php_mssql driver
3. MSFT SQL Driver ( See pcorbetts posting for a link to the driver )

PHP 5.2.5 breaks support for SQL Server
I'm running PHP 5.2.5. I've had a perfectly good working SQL Server installtion, so I thought this was just a PHP upgrade and then I was going to continue to do some module dev. and bug fixing. Well, it seems that with PHP 5.2.5 neither the php_mssql extension OR the FreeTDS driver work with PHP any longer. I have submitted a bug report to PHP ( 43645 ). This has been verified by another developer also.
MSFT SQL Driver works in 5.2.5...but still alpha
The MSFT SQL Driver DOES work in 5.2.5. However, the driver is still in alpha quality, and will prob. be Spring before I would say it's a stable beta ( reference the IIS FastCGI module for a MSFT timeline :) ).

It should also be noted that the ntwlib client ( this is the library that allows FreeTDS and the php_mssql libs to connect nativley to SQL Server using TDS [Tabular Data Stream]) is no longer going to be distributed with SQL Server. It's been deprecated. This kind of makes sense, given MSFT is prob. going to start branching it's product away from it's Sybase origins, so who knows what funky stuff they'll do with the TDS format. Anyway, I'm still trying to get my head around what direction they're going to take at this juncture. They must know there's a need for PHP+MSSQL connectivity, thus the development of the SQL Driver. I know that it is beta ( and as I've said before...definatly a TRUE beta, not a "google" beta :) ), but I'm confident this driver will get finished and supported...but I just can't say when.

With that said, I'm going to try and continue development by downgrading my PHP, and keeping an eye on the but I opened at php.net.
That's all for now, but just thought I'd give a heads up about PHP+MSSQL and esp. the 5.2.5 issue.

Enterprise

Active Records, a possible approach for consistent Data APIs

recidive@drupal.org — Thu, 25 Oct 2007 23:53:54 +0000

As mentioned on the other paper A Data API for Drupal. Here is a paper that shows how this could be accomplished with some OOP bits implementing the Active Records Pattern. While keeping the interface procedural.

We all know that Dupal will not go full OOP. But some OOP code could help us do things not possible with procedural code. Such as lazy loading of nested objects and 'on demand' database fields processing.

Before going this route in Drupal core we would need to rigorously benchmark and carefully weigh the risks and benefits. Record module is intended as a step towards enabling us to thoroughly evaluate these options.

Services

Data APIs for Drupal 7 and web services support

nedjo — Thu, 25 Oct 2007 19:21:19 +0000

There's growing interest in the Drupal community in the prospect of renewing our core data handling APIs. Doing so will increase consistency and efficiency and ease barriers. It will also be a key step in enabling transactional web services.

In Drupal 6 we took some impressive first steps. What should we tackle for Drupal 7?

The attached paper, written by Nedjo Rogers and Henrique Recidive and sponsored by CivicSpace, aims to carry this discussion forward and map out both some conceptual space and concrete development tasks.

An accompanying post will present for review and discussion a proposed Active Records implementation for Drupal, taking advantage of features built into PHP 5.

Please, wade in and review, critique, and improve!

(I'm posting this to the Schema API and Services groups as the two most closely matching this topic. We could consider forming a new Data API group as well.)

Services

Table creation order

bjaspan — Tue, 12 Jun 2007 01:48:56 +0000

Once foreign keys are specified during table creation, we will have to make sure to create tables in the correct order. e.g. You can't create system.schema, whose files table has a uid column, before you create user.schema which creates the users table which hold the master uid column.

If this proves too difficult we can use ALTER TABLE to put the foreign key contraints in afterward, but that seems much less desireable.

Database

Add join/foreign key info to schema

bjaspan — Fri, 01 Jun 2007 17:31:12 +0000

Now that we have four more weeks before D6 code freeze, maybe it is possible to add join relationships/foreign key support into Schema API 1 after all. Otherwise, it will be another year before this info is in core. Note that this info does not really have to be in core---it can be added by a contrib module hook_schema_alter. But a lot of people would be pretty psyched to have referential integrity for pgsql built in to D6. So:

Make a list of all the key relationships in the core tables.
Decide how to encode the info in the schema data structure.
Add foreign key constraints in database.pgsql.inc.
...
Profit!

I have a pretty good idea for all the steps (at least the first three :-) but, for right now, am just creating this post as a placeholder. Jump in!

Database

Additions to the API

drewish — Mon, 28 May 2007 19:23:34 +0000

Now that the schema has been committed to core, I wanted to open up a thread where people could discuss additions an enhancements to the API.

I posted a patch to add a db_rename_table over on #147285. I've love to see it get in before the feature freeze.

Database

Defining Schema data types

bjaspan — Tue, 24 Apr 2007 13:35:57 +0000

Now that Schema almost has a driver for PostgreSQL I needed to resolve the data type issue. I like Frando's approach (I seem to say that a lot) of storing 'type' and 'size' separately because it makes a variety of things easier. For example:

$schema['mytable'] = array(
  'cols' => array(
     'col1' = array('type' => 'int', 'size' => 'big', ...)));

In a column specifier, 'size' is optional and defaults to 'normal'. For type varchar, 'size' is ignored. I changed a few of the constants Frando defined to make them more like what I think people will expect (e.g. 'varchar' instead of 'string') and I normalized the size constants so that all types use 'big' instead of some using 'long'.

So, what I have so far for MySQL:

    $map = array(
      'varchar:normal' => 'VARCHAR',
      
      'text:small' => 'SMALLTEXT',
      'text:medium' => 'MEDIUMTEXT',
      'text:big' =>  'LONGTEXT',
      'text:normal' => 'TEXT',

      'int:tiny' => 'TINYINT',
      'int:small' => 'SMALLINT',
      'int:medium' => 'MEDIUMINT',
      'int:big' => 'BIGINT',
      'int:normal => 'INT',

      'float:tiny' => 'FLOAT',
      'float:small' => 'FLOAT',
      'float:medium' => 'FLOAT',
      'float:big' => 'DOUBLE',
      'float:normal' => 'FLOAT',

      'blob:big' => 'LONGBLOG',
      'blob:normal' => 'BLOB',

      'datetime:normal' => 'DATETIME',
    );

and for PostgreSQL:

    $map = array(
      'varchar:normal' => 'varchar',
      
      'text:small' => 'text',
      'text:medium' => 'text',
      'text:big' => 'text',
      'text:normal' => 'text',
      
      'int:tiny' => 'smallint',
      'int:small' => 'smallint',
      'int:medium' => 'integer',
      'int:big' => 'bigint',
      'int:normal' => 'integer',
      
      'float:tiny' => 'real',
      'float:small' => 'real',
      'float:medium' => 'real',
      'float:big' => 'double precision',
      'float:normal' => 'real',
      
      'blob:big' => 'bytea',
      'blob:normal' => 'bytea',

      'datetime:normal' => 'timestamp',
      );

Comments?

Database

Random ID Generation Write up

Souvent22@drupal.org — Tue, 24 Apr 2007 02:58:43 +0000

So, I've had a few inquiries over the last few months about random node generation as I have implemented it successfully for a client. I have posted a full write up of it on my blog. Feel free to comment on it here or on my blog; I will post a patch when I have a chance and if there is interest.

Article link: http://earnestberry.com/node/13

Coding Standards and Performance Optimization

Using database schema to load nodes in fewer queries

bjaspan — Tue, 17 Apr 2007 04:23:23 +0000

One thing that becomes possible when Drupal can access its own database schema is that queries which currently are performed separately can instead be performed together. In particularly, I think it might be possible to load an entire node with a single SELECT query or, at least, in far fewer than Drupal currently uses. Working proof of concept code is already in the Schema module; it loads all CCK fields, comments, and author info in one query. Loading more things only requires adding a join entry to the schema data structure (see below).

Note that I do not yet know if it is faster to load nodes in one query, only that it seems possible. I'd like to use this thread to explore the idea and see what people think. To get started, I'm reposting here (with some updates) a followup I posted in http://drupal.org/node/136171.

Suppose I define a CCK type called "type_1" with a single field, "field_1". In Schema's data structure, the table looks like this:

$schema['content_type_type_1'] = array(
    'cols' => array(
         'vid' => array('type' => 'int', ... 'join' => array('node', 'vid', 'one')),
         'nid' => array('type' => 'int', ...),
         'field_field_1_value' => array('type' => 'longtext', ...)),
    'keys' => array(
         'PRIMARY' => array('vid')),
);

(Actually, that isn't exactly how it currently looks; the data structure layout is a moving target.)

So, the table has two columns, vid and nid, and its primary key is vid. The vid field is a 1-to-1 join to node.vid.

When a node is loaded, Schema's hook_nodeapi is called. It looks through the total schema (the sum of all structures like above returned by all modules) to find all joins to node (because this is nodeapi, we know we're loading from table node) and builds up a SELECT/JOIN list. Suppose the schema contains type_1 as above and another module with a table m2 that joins to node on nid and has columns 'foo' and 'bar'. schema_nodeapi builds this query:

SELECT content_type_type_1.nid as content_type_type_1_nid, m2.vid as m2_vid, m2.foo as m2_foo, m2.bar as m2_bar
FROM node n
LEFT JOIN content_type_type_1 ON content_type_type_1.vid = v.nid
LEFT JOIN m2 ON m2.nid = n.nid
WHERE n.nid = %d

It executes this query for the nid being loaded and saves the results of this query in $node->schema_data. schema_data['content_type_type_1'] contains the array of matching rows from the content_type_type_1 table and schema_data['m2'] contains the array of matching rows from the m2 table. So now, CCK's nodeapi and m2's nodeapi do not need to execute their own SELECT query. They can both just use $node->schema_data which is waiting for them. So, we've turned three queries into one.

It gets a little more complex if one or more of the joins are 1-to-many (e.g. comments, multi-valued CCK fields, taxonomy terms, etc.). However, so long as the joined tables have a primary key, it is possible to identify all of the unique rows of data returned from each table and save them in $node->schema_data, ignoring the duplicates that come as the result of multiple multi-value joins. (This code is written and working too but not yet committed as it needs some cleanup first.)

Currently, this happens in schema_nodeapi() but it obviously could be moved directly into node_load() as part of the initial SELECT into the node table, eliminating another query.

Schema exports $schema structures for all CCK types and fields so this automatically works for all CCK per-content-type fields in the content_type_* tables and multi-valued or shared per-field fields in content_field_* tables. Depending on the relative performance of this one query with a bunch of LEFT JOINs, we might be able to eliminate CCK's field cache table, cutting the data storage requirements for CCK nodes in half without a performance penalty.

And this isn't limited to CCK. As I mentioned above, everything that connects to node could be loaded at once: comments & comment statistics, taxonomy terms, the author's user record, and all the add-on data currently loaded by many modules (book, event, etc.) that is keyed on nid.

This does result in a lot of LEFT JOINs. One optimization is to use node types to eliminate unnecessary joins. For any given node, only a subset of tables that join to node.nid/vid are relevant: content_type_page is only relevant for page nodes, and the book table is only relevant for those node types that book.module is configured to book-enable. Each module could encode in its schema the node types the join is relevant for. For example, if book.module is only enabled for 'page' nodes, it could say:

function book_schema() {
$schema['book'] = array(
'cols' => array(..., 'join' => array('node', 'nid', 'one'), 'node-types' => array('page'))
);
return $schema;
}

Schema's nodeapi knows the type of the node being loaded because node_load already got it, so it can simply skip any join that specifies a node-type list and doesn't match the current node's type. To move this one-query approach into node_load and use this optimization, we will need to keep a cache mapping nid => node type (note that a node, once created, never changes its type). Since node types are write-only, this could be done as simply as having cron output a file defining a array containing the mapping, so no query is even needed to get the nid => node type cache, just a file load. Schema already implements the node-type optimization but I have not yet implemented the nid=>type cache.

So, this is interesting and all, but the most important question is: Is it faster? I honestly don't know. Maybe one big query with many LEFT JOINs is slower than 10-15 separate queries, each against one or two tables. There is also some overhead in processing the results of the big query, ignoring duplicate rows, etc. Performance testing will be necessary.

I'm also quite sure there are bugs to be worked out, improvements to to made, etc.

Questions? Feedback?

Database

Defining the schema data structure

bjaspan — Mon, 16 Apr 2007 20:59:18 +0000

One of the first things we to do is define the data structure that modules will use for declaring their tables. Schema module, Frando's patch, dopry's patch, and CCK fields all use different structures though their underlying approaches are basically the same. We need to decide what kind of information the data structures must contain and can contain. This will obviously include at least columns (name, type, length, nullable, default), indices (name, columns, primary vs unique vs multi), and join/relationship information.

We should be database-independent enough to support a broad range of DBMS's but not try to be so general that we go crazy. The vast majority of Drupal uses int, varchar, and longtext columns, with a float here and there. We don't have to support everything.

Defining the join relationships is critical. Eliminating CREATE TABLE is a primary goal, of course, but I believe the functionality enabled by having join information available will be the killer app for the Schema API.

So, the floor is open. What should the table-definition data structure look like?

Database