Budapest internationalization meeting findings

Drupal 5.x and before has a "locale" concept, which lets a user see the interface of the website in a choosen language. The list of available languages for locale is defined by the administrator, and a user can choose a preferred language on the user profile editing page. It is only possible to translate the built in interface text though, and none of the admin specified text (ie. site name, site slogan, menus, categories, blocks, etc) is translateable. It is also not possible to translate content sent in by different users of the website.

Some solutions evolved through the years to overcome this limitation, making it possible to translate user defined interface and content elements. Unfortunately these modules only build on what is available in Drupal and are not as closely tied as some performance requirements or design considerations would require. Since the Drupal core code is not ready to handle this situation, most contrib modules don't care about offering this feature, and internationalization (i18n) modules need to work around them too.

In getting ready to add support into Drupal 6 for better internationalization, we examined the code base of the available modules (although these are actively changing), identified the problems, looked at the possible outcome of some development around custom node types, menus, etc. The summary bellow is a list of items we worked on last week in Budapest and reasoning behind some implementation details.

Try the code! Suggest better implementations!

You can try the code by grabbing our (temporary) Drupal fork from the Development Seed sponsored SVN repository. As we have the Drupal CVS metafiles comitted too, you can do a cvs diff on the codebase to see what did we change from the Drupal 6-dev. And because what you get from the SVN repository is a complete Drupal core, you can install it and try it out. We did not get around yet to provide upgrade paths, so do not try this code yet on a previously used Drupal database. Do a fresh install!

Language management

In Drupal 5.x the langauge list is managed by the locale module, and the attributes of languages is restricted to those needed by that module. I18n related contrib modules reuse this list of languages, some adding more properties. We found that Drupal core itself would need more properties for each language, so that themes can be made to support different language behaviour. After discussing this with Dries Buytaert, we decided to keep the language management functionality in locale module, but it is now on a separate primary task.

What changed?

• "locales_meta" table becomes "languages"
• The "locale" key name becomes "language" (also in "locales_source" and "locales_target" tables)
• The "languages" table gets a new "direction" field, so we can store "Left to right" (LTR) and "Right to left" (RTL) directionality information, which is even needed by core themes to adapt to RTL languages. The built in language list gets new direction values, so automatically added languages have this value properly set. [TODO]
• The "languages" table gets a new "native" field, which stores the native name of the language to show on language switchers. This is important for those who visit a foreign language page, since we can help them spot their language with using the native name (too). The built in language list already has a list of native lanuage names.
• The "languages" table gets a "weight" field, so languages can be weighted. This can be used for language negotiation (but is not used for this (yet?)), and can be used for listing languages by weight for language selection.
• The "languages" table gets a "prefix" column, where a path prefix can be specified for that language.
• The "languages" table gets a "domain" column, where a language domain can be specified for that language (eg. de.example.com or example.de).
• The "languages" table looses the "isdefault" column, in favor of storing the serialized default language object in the variables table. More below.
• The language management screen is now a standalone primary task, with listing and addition as secondary tasks. The locale import/export functionality is a secondary task of the locale functionality.

Performance

The language list is needed by locale module and is also needed by any extending functionality to handle languages on Drupal objects (nodes, taxonomy, blocks, etc). The language list is not needed on a site where there is only one language used (English by default, but can be set to a different language). For this reason, we introduced two variables: "language_count" and "language_default". We always store the number of languages set up into "language_count", and if it is only one, we consider "language_default" to be that one, and use that exclusively. This means that we can completely sidestep language detection and other language management code when serving pages on a single language website, as these two values are directly loaded with other variables anyway.

Impact, future usage

The user sees one single spot for setting up languages. Since languages have their own forms, any extending functionality can hook into them (see hook_form_alter()) and add more settings to languages. Having the weight and direction information, themes can adapt to finer language detail, like adding the appropriate HTML attributes when handling an RTL language.

Language negotiation

Once a site has more than one language to show interface and/or content in, somehow the used language or languages needs to be choosen. We have decided to go with a single language selection for now with fallback to the default language if possible, but there are valid use cases for more complicated sites where a list of possible languages (eg. for content filtering) is desired instead of a single language.

What changed?

Because the decision on the language used has an impact of the variables being loaded, the cache being checked and so on, it was obvious, that a bootstrap step should be added to check for the language, and decide on the language used. We examined the existing code used for page caching, and found that it is ready for dealing with domain and path prefix changes, so to not hurt cache served page performance, we added language negotiation right after the "late page cache" bootstrap phase as a new phase.

The $locale global variable become $language and is now not only a language code, but a full language object, which contains information about the native name, the direction and other attributes of the language. This is directly passed to phptemplate themes too.

We added language.inc to get loaded when multiple languages need to be handled. Language negotiation (the selection of the displayed language) and path handling is now implemented there.

How does it work?

1. If there is only one language set up, the default language is used always.
2. If there are multiple languages set up, the admin can choose one negotiation scheme.
   1. No selection. This always chooses the default language, and is the default scheme. It is handy to keep previous practice (ie. you don't suddenly change your URLs when you start to set up languages).
   2. Domain based. If a language specific domain is found (arbitrary domains can be set per language), that language is used, otherwise the default. This scheme could have paths like http://example.de/node/1 and http://example.co.uk/node/12
   3. Path based with empty default. All languages use the same domain, but different path prefixes. The dafault language has no prefix, so if a path without language prefix is identified, the default language is selected, otherwise the requested language. This scheme could have paths like http://example.com/node/1 (default language) and http://example.com/de/node/14 (German).
   4. Path based with default as redirector. All languages use the same domain, but different path prefixes. Every language should have a prefix. If a path without a language prefix is found (eg http://example.com/) a negotiation algorithm runs. It tries to check for the user language (set on the user profile page), then falls back to the browser preferences, then falls back to the default language if others fail.

   Other schemes might get implemented, or we might add an extension point for other modules to implement additional fallbacks like IP-to-county based language detection. [TODO]

Path aliases

Once we have available languages and a language choosen, we can provide language based path aliases, if available. This is important to find the right menu callback to fire, so this takes place right in bootstrap.

What changed?

drupal_get_normal_path(), drupal_get_path_alias() and drupal_lookup_path() all got a new optional $lang parameter, which is a language code to look up the path for. This way drupal_init_path() inits the path with a language specific alias, if available. To store language specific aliases, we introduced a "language" column in the "url_alias" table, and a language selection box on the alias editing/addition page.

Performance

Our current implementation has a (possibly small, not benchmarked yet) performance impact on users without language specific aliases. Ideas are welcome to improve the situation.

Variables (ie. site settings)

The inherent problem with variables is that modules provide all kinds of interfaces to set them. Some variables are computed, some are not available on the user interface to set, and some are set with exotic interfaces (with possible interdependencies, etc). All we have been able to agree on was that we need to provide language dependent site settings, and if we do it in Drupal core, we can better cache what is expected to be required by the page, so we get better performance for served pages. How the user interface will look is not finalized yet. [TODO]

What changed?

variable_init(), variable_get(), variable_set() and variable_del() got a new language parameter, which is a language code to work with. We introduced per-language variable caching, and we added code to be able to load variables for any desired language, and then perform a fallback on the default, if the requested variable is not available.

This part of the code still needs some polishing, but the general idea is there. [TODO]

Nodes

We discussed nodes to great length, especially in the light of some user defined field code possibly getting into Drupal core.

Problems with "inline" node translation

A few people noted before that we should support node field level translation, and should not store translations of nodes in different node instances. For that to work, basically all node properties should be a field (ie capable of being replaced by a different value, when translated). That means that from the perspective of i18n we should make taxonomy terms, uploaded files, author ids, submitted dates, titles, bodies, node level properties like published, promoted, etc. "replaceable fields" of the node. So when a translation comes along, we can associate different pieces to the translated node.

What would remain constant in all cases, is the node id. In case we would like to have this perform well, we would need a powerful query rewriting mechanism, where we can request an author id, published flag, taxonomy term association, etc. with a language condition for a node id, so that unused data will not get queried, and we don't need to have multiple queries for a node to grab different fields from different tables.

Still then the node id would remain, which is what node level permissions connect to (additionaly to taxonomy terms, uploads etc), so permissions should also be made aware that a node id is not a unique content id anymore.

It seems that if we would like to implement this with good performance in mind, we would need to overhaul the node concept and tack language into every aspect of it. This would probably not get a warm welcome from those who are not interested in multilanguage sites.

If you have better ideas here, speak up!

Translation sets to the rescue?

Because with languages, we are basically adding another layer on top of nodes anyway, why not do the implementation from this perspective? With earlier implementations, the problem was that 'nid-language-nid' relations were stored. However, if we introduce the concept of "translation sets", we can group translations of the same content into one set, and give that set an ID. Either this set is itself a node, and contains the common properties of all items in the set, which are projected to the nodes when being edited/displayed/queried, or we need to be able to mark shared fields in the set member nodes themselfs. Either way, we would probably need to copy the values over to the set member nodes when being saved, so we don't need to hit the database with the node mashup algorithm on every query. This is why it is probably better to not have the set as a node, since that node would be a rather thin one in most cases, actually making it unusable.

Menus, taxonomy, blocks, profiles

These are still to be looked at more closely implementation-wise. Blocks and profiles are expected to be simpler, since we do not intend to provide capabilities for users to translate their profiles to different languages, just administrators to translate the field names and descriptions to different languages. Blocks are very similar.

Menus and taxonomy are a whole different question. Some people like to see different menu and taxonomy trees for different languages. On a website, where only a few pages are translated, the translated versions will have a lot less menu items and a definitely smaller taxonomy term list to work with, since listing hundreds of empty pages for taxonomy terms nonexistent for that language is simply not an option.

For those sites, where all taxonomy terms have equivalents in other languages, a different UI option can be presented so that it is easier to edit taxonomy terms and translations.

Expected future work we intend to build on

Some of the future work possibly in the Drupal 6 timeframe which affects our work:

1. Menu system. The user defined menu item system is under rework, so we are awaiting what we can work with.
2. SQL query builder. There is word on the street that a better (not preg dependent) SQL query builder is on the way. We look forward to using that for query rewrites.
3. User defined fields. This should open up a whole new can of worms, so we are very interested.