Report of eight months of work on Drupal 8's multilingual improvements

It is hard to believe but in a few days it is going to be eight months, since the Drupal 8 Multilingual Initiative (D8MI) was announced. So it is at least as good a time as ever to take a look back and take stock of where we are and where we need to accelerate work to get to results. On one hand, the D8MI is very lucky for being able to do work on many fronts at once and make small incremental changes instead of turning Drupal upside down (making the changes easier to accept). On the other hand, sometimes the changes themselves don't make a big impact and only form a bigger picture when considered all at once and more importantly depend on some bigger changes being done in the entity system or the configuration system, which are big dependencies.

WARNING: In Dries Buytaert's latest published words, Drupal 8 is about to be released in 17 to 23 months from now. (The interview is dated mid-last month and cites 1.5/2 years). All the new stuff detailed here is to be released in a stable Drupal version at that time with a huge number of other improvements. Drupal 8 is not in any way near release.

A three-legged approach

Instead of turning multilingual concepts upside down, D8MI attempts to make the classic three legged multilingual approach in Drupal work much better and more unified, taking best practices from each area. The three legs are interface translation, content language/translation and configuration language/translation. The differences for these come from deep Drupal implementation differences.

1. The built-in software interface is written in English, can be pre-translated by the community and distributed to Drupal installs. The community translation itself is out of the scope of Drupal core, so we need to improve support for the distribution/access to translations itself.

2. The content translation system uses entities and fields, which are great because it encapsulates data storage with the widgets and formats/validation of the data. This is really good for language support / translation, so we are planning to expand this pattern to all entity types and get the user interface for translation in core.

3. For configuration (everything that is not entities or the built-in user interface), Drupal 8 is slated to include a configuration management backend, that is in the works and not yet well defined on the user interface level. This continues to be both the biggest feature gap and biggest unknown for the D8MI.

Base language system

Bigger focus on language in the installer

We moved the language selection step to be the very first step in the installer from which every screen can/will be translatable. This step now displays a more compact list of languages when many are available and also detects your preferred language based on your browser preferences if available. This means that even foreign language installs could run with just one click through the language selection screen to verify the preselected default. We have some ideas to improve the installer even more, see http://drupal.org/node/1337628.

An improved language system

To build a common solid basis for the above mentioned three legs and finally provide a clear and familiar language API, we introduced a distinct language.module in Drupal 8 that provides language management services to be used by all three legs. This module provides new clear APIs for simple operations like language_load(), language_save(), language_delete() and corresponding hooks, like hook_language_insert() and hook_language_update() among others.

We have also made big improvements in the language detection system, especially in browser preference handling. See http://drupal.org/node/221712 for details. It currently looks likely that language negotiation moves in with the basic language API as well after an API cleanup. More discussion is going on at http://drupal.org/node/1293304.

Simplified and unified language management

We have simplified adding languages (you only need to provide a language code, name and direction even for custom languages), and made the overview of languages on your site more useful by integrating live data from modules related to language, such as your interface translation status (which directly link to translation forms). This information is pluggable, so other modules can provide information here. We expect content language/translation to be among the features represented here in the future.

English is not (much) more equal than others

New in Drupal 8 again, you can delete English from your system if you don't need it (see the delete link on the langauge list screenshot), moving it completely out of your way. If you do need English but the built-in user interface text is not entirely fit for your needs, you don't need to add a custom English just to translate the interface anymore. You can just go edit English and tick a checkbox to be able to translate Drupal to English. The checkbox is purely there for performance reasons, so if you don't want to translate Drupal to English, there is no performance impact of this new feature.

Better in-context negotiation configuration

While we are still working on making negotiation self-configure itself so you don't even need to think about it unless the defaults are not a good fit, we made certain parts of negotiation configuration much more useful. If you go configure URL base negotiation for example, you'll get in-context configuration forms for domains or path prefixes based on what you need. You can see and compare all settings at once. By moving this configuration from the language forms we both made language setup simpler and negotiation configuration easier I hope.

HTML 5 compatible language markup

We also worked with the HTML 5 initiative to fix the page markup to use HTML 5 compatible language attributes (removing xml:lang), provide node level language information when separate from the page language (great for pages displaying multilingual content) and made language configuration more straightforward by adopting HTML 5's language code standards (see the custom language configuration screenshot above).

Interface translation

We made some great low level API improvements for interface translation. The security/placeholder features of t() are now available in a separate function as well that does not do translation called format_string(). Great for those cases when you need to t() something and do the replacements later. Drupal 7 introduced context for t() strings, but this functionality was not ported to the Javascript API. Drupal 8 introduces context support for Drupal.t() and Drupal.formatPlural() alike.

Drupal 8 also introduced a new gettext.inc file in the includes directory that houses low level Gettext .po file parsing and creation routines. This was included with locale.inc in Drupal 7 and unnecessarily loaded on multilingual pages. There is further work planned to untangle this code entirely from the interface translation API and make it pluggable (as well as support batch API based chunked file reading/imports to work better on constrained environments). See http://drupal.org/node/1219236 to be part of this work.

Earlier versions of Drupal provided a semi-API for different textgroups in the interface translation system to implement configuration translation reusing this system. With Drupal 8 generalizing gettext parsing and the interface translation storage proved to be a bad place to store configuration translation for various architectural reasons, Drupal 8 phases this out, separating this leg from configuration translation clearly. This also noticeably simplifies the interface translation forms.

While earlier Drupal versions were looking for .po files in /translations directories under each module/theme/profile used, Drupal 8 centralized these to one directory. This is better for both deployment and staging systems as well as more compatible with version control and easier to make secure. This is a precursor to integrate the Localization update module, for which there are many other outstanding issues. That is still a bit far off, so help more than welcome at http://drupal.org/node/1329410.

Finally, by integrating the interface translation status in the language list, the admin interface now provides much more direct access to the translation forms, either form the language list as mentioned above or right in Regional settings. The interface translation UI is not hidden under a tab anymore.

Content language/translation

There has been great ground work in this area mostly outside of the Drupal 8 Multilingual Initiative. Entities are now managed by entity.module and entities are getting object oriented interfaces. There are still lots of work ahead of us in this area, but the OOP conversions of entities would highly conflict with any langauge work, so right now helping with the conversions would be of great support for D8MI. See http://drupal.org/node/1346204

The best place to help directly at the moment however is to read the writeup for content language plans at http://groups.drupal.org/node/197848 and provide your feedback there. The user interface discussion for content translation is going on at http://drupal.org/node/1282018 which should be also of particular interest.

Configuration language/translation

The inevitable dependency for this leg is the configuration management initiative's API work. We've been in discussions on levels and solutions for language support and work is ongoing in introducing a configuration management API. Regarding its language aspects the architecture document details some ideas at http://groups.drupal.org/node/191283 and there is a separate discussion at http://groups.drupal.org/node/185609. Please participate in these discussions to help shape what's available and how.

In summary

As you can see we made lots of progress on the base level language support area but bigger user problems are still to be covered. You can help a lot clearing the ground for D8MI by helping with the entity conversion work and the configuration management initiative's initial APIs so we can build forward from there.

We are holding a Drupal 8 Multilingual Initiative meeting later today in the #drupal-i18n channel and attempt to set up a pratice of repeating these events every two weeks if there is interest. Be there, help with more great improvements! Let's make Drupal 8 the most rocking version for the world at large!