Drupal 8 Multilingual Sprint report from Barcelona

We encourage users to post events happening in the community to the community events group on https://www.drupal.org.
Gábor Hojtsy's picture

Just this past week, we had a full week of sprinting on Drupal 8 Multilingual issues on ocassion of the Drupal Dev Days over the weekend. Citilab, the venue for the event was generous to let us use the space for the whole week, and four fantastic financial sponsors, dotProjects.be, Cando Image, Open8, Acquia and Dave Reid made it possible for the sprinters to have a worry-free week. Their sponsorship let us have our coffee, soft drinks, lunches, ice cream, fruits and vegetable snacks and even a very nice dinner downtown covered, so we can just focus on what we are good at: solving multilingual issues for Drupal 8. We absolutely grateful for their support.

We have had about ten people starting off the week sprinting which grew up to over 20 by Friday. Key people like Internationalization module maintainers Jose Reyero (reyero) and Florian Weber (webflo), Localization update maintainer Erik Stielstra (Sutharsan), Entity API lead Wolfgang Ziegler (fago), CMI initiative lead Greg Dunlap (heyrocker), core generalist Daniel Kudwien (sun), core committer Angie Byron (webchick) were around, which really helped us make great advances. Amazee Labs really put the multi in the sprint by sending not one but two great developers, Micheal Schmid (Schnitzel) and Vasi Chindris (vasi1186). We've had long time hard workers Peter Droogmans (attiks), Christian López (penyaskito) and Clemens Tolboom (Clemens.Tolboom, remote) in attendace and even Francesco Placella (plach, remote) could contribute all the way from Venice. We also had people helping us test our new user interfaces and figure out problems, including Lowell Montgomery (LoMo). Various people were sprinting focused on Entity API, including but not limited to Sebastian Siemssen (fubhy), Sascha Grossenbacher (berdir), Miro Dietiker (miro_dietiker) and Frederic Marand (fgm, OSINet). Thanks again everybody for coming and participating! (And I'm sorry if I forgot to mention your name). And special thanks for our on-site organizers, Pedro Cambra and Ramon Vilar for their amazing help.

We have made great advances in all four areas of the Drupal 8 Multilingual Initiative (base language, interface translation, content language and config language).

Key results already landed in Drupal 8

All language globals are now gone

The process of removing language globals is now complete. The Dependency Injection introduction that was started with the language negotiation values in http://drupal.org/node/1497230 is now the sole way to access negotiated language values for the page. See the change notice at http://drupal.org/node/1539454.

Language class is now used universally

The Language class that was introduced with the Dependency Injection process is now used universally with language_list(), language_load(), language_default(), etc. This means you can use type hinting for parameters that are supposed to be language objects. Further work to make the Language class full featured is being discussed at http://drupal.org/node/1512424

Introduced locked languages, more special languages exposed

We broke down the Drupal 7 'Language neutral' language to 'Language not applicable' and 'Language not specified' earlier and added 'Multiple' as language constants, but we did not have a UI to actually assign these languages to things. Now, you have these languages show up on your language list with explanation of their roles and you can reorder them and apply them to content, etc. See http://drupal.org/node/1471432 for the issue introducing this change.

Locale module is now purely for interface translation

In Drupal 6 and 7, Locale module has gotten fatter and fatter with code to solve language problems on behalf of other modules. Most of these 'solutions' however were mainly implemented in the respective modules (such as for node language, the language storage, filtering, administration was all in node module, only a little piece of UI was in locale module). In Drupal 8, our approach is that we don't attempt to solve language problems from the outside, so node, path, etc. related code now resides in the respective modules. On last week's sprint, we got the remaining two pieces, localized date formats and field language moves to their respective modules, leaving locale module purely for interface translation (given the base language functionality is in Language module). This should let us focus this module only on this functionaliy and clean it up as such. The two issues related were http://drupal.org/node/1630264 and http://drupal.org/node/1635134.

Brand new interface translation translation interface

Drupal 7's built-in interface translation interface is a flash of the past, where you likely only had your Drupal sites in a couple languages that you all knew and you wanted to translate things all at once to multiple languages. That is not really the major use case anymore. So we took a fresh look at the translation interface and rebuilt it from the ground up. Now it focuses the user on one language and lets users translate multiple strings at once. It also builds-in singular/plural string pairing into this user interface, that was not even possible on Drupal 7. User reaction so far has been overwhelmingly positive. We've been working on this at http://drupal.org/node/1452188

New, highly flexible node type language defaults setup

Drupal 7 has language support for node types hidden under 'Publishing options'. There are three options for node language support. There is 'no language support' in which case, nodes are saved as 'Language neutral' and a language selector never shows. There is 'language support enabled' in which case the node will show a language selector and default it to the site language and this can be extended with 'translation support', which lets users translate, if the node had a specified language. This is extended a great deal in Internationalization module, but the setup is not very clear.

In Drupal 8, you now find language setup under a dedicated vertical tab. You also now have a built-in language default configuration user interface that is a lot more flexible. Now you can separately set the default language and whether to show/hdie the language selector, so you can set up configurations like default all your forum posts to a specific language or default node submission language to the user's preferred language instead of the site's. We are working to generalize this to the entity level, so you could have the same configuration on taxonomy terms for example in the future. Our issue for this work was: http://drupal.org/node/258785

Drupal now tracks the status of your imported .po files

Drupal 8 already centralized .po file lookups to one directory, so you don't need to hunt around for your .po files under different modules and themes. Work is ongoing to build-in the functionality of Localization update module, so that your site can be fed with interface translations automatically from localize.drupal.org. Yet another step of that integration is now in core! Locale module will now track all files that it previously automatically imported, and will only import them again if the date of the files changed. This is purely backend functionality, and the rest of localization update integration will put a simple user interface on top of it. We've been working on this in http://drupal.org/node/1635084

Discussions and ongoing work

Configuration management with multilingual configuration

We've had some great and extensive discussions leading up to the sprint with key people on the requirements and the design for a multilingual configuration solution for Drupal 8. We have mapped out an option A (with variants AB and AA in discussion there), option B and option C. The various options used various storage models, call patterns, context handling, and so on. I've made this figure to summarize the situation and we had various meetings to discuss the options with Earl Miles, Greg Dunlap, Jose Reyero, Daniel Kudwien, etc. Going through what we need and how we can get that done, option B won out, with the caveat of working together with the context/plugins systems introduced in Drupal elsewhere. Read the whole proposal at http://drupal.org/node/1616594.

  1. Translatable vs. multilingual: Our discussion evolved around supporting both translated (like site name) and multilingusl (like default front page path) values, which differ in conceptual approach to whether they are human language translation or just value storage keyed by language. All other options but option B only allowed for storing some values for different languages and would have caused developer experience issues.
  2. Config storage: To potentially allow any configuration to be multilingual, we either need to change the structure of configuration on the fly, introduce language at all places (like field API did, and freaked developers out), or externalize language variants to the degree that they are still managed within CMI storage for staging and deployment but don't decrease the developer experience. Option B externalizes language storage to separate 'override' files for languages (and optionally allows for other realms to have different values).
  3. Context: We agreed that for storage and retrieval we need some kind of context. If you just think about field settings, we need to (a) display an admin form for them in their original language, (b) display them as labels on the entity form in the interface langugae and (b) display them as content data in the content language on entity view. So we really need developers deal with some kind of language context either way, indepdent of which method we choose. Option B passes this context around without requiring each method and function to get a specific language argument.

We have also been busy helping to port some settings to CMI to serve as our use cases, such as site information (http://drupal.org/node/1496542) and contact form settings (http://drupal.org/node/1588422). Neither of those landed yet and needs help.

We've also attempted to convert the built-in language list to CMI for distribution, which sparked some debate as to whether this is even configuration at the first place. You opinion welcome on http://drupal.org/node/1632236.

Entity property translation

A cornerstone of the Drupal 8 Multilingual Initiative is to reduce the two entirely different options for content translation to one. Instead of node-copy based translations and in-entity field translations co-existing, we want to extend in-entity field translations to support all fields and properties and drop node-copy based translations (migrating your data to the new model). For this, the Entity API work convering property API and making all properties use similar storage to fields is key. Turns out this is a shared interest between us and the WSCCI initiative, and there was extensive discussion about the topic at the WSCCI Web Services Format sprint, so we got some great looking plans from there. People sprinted on making this happen, but it is far from ready and needs lots of hands. Let me know if you want to help!

There has already been considerable work to convert some accessors to support language, like the label() method on nodes and properties, see http://drupal.org/node/1616952, http://drupal.org/node/1616962 and http://drupal.org/node/1616972.

Gettext parser refactoring

Clemens Tolboom has been hard at work on this before, now joined by Peter Droogmans and Christian López, they worked out a great interface and class structure to make out Gettext .po file parsing and generation up to date. Our current code in Drupal 7 is as coupled as it can be, taking a file name and directly inserting to the database of locale module all at once. The refactoring work makes it possible to take input from other file formats or export to other file formats, and just generally makes the parser and generator up to date to current standards. There is still work to do on this front (at http://drupal.org/node/1189184), and we created this figure to show the structure of the parser to those interested:

Integrating project awareness to locale module from localization update

Erik Stielstra and Florian Weber were busy to integate more of localization update in core. While file status monitoring landed (see above), work is ongoing to make project identification and file download happen. Erik worked out lots of new test cases, that were not at all in the contributed module to prove everything works right, and he needs more hands to continue this work. See http://drupal.org/node/1627006 (project identification) and http://drupal.org/node/1632384 (project file imports).

Language-aware entity form controller

We've also been continuing working to introduce language awareness to entity forms, so we can dislay the same entity form in different languages for translation. See http://drupal.org/node/1499596 for more information.

Gearing up for Munich

Drupalcon Munich is very close! In just two months from today, people will gather in Munich to celebrate Drupal's greatness, learn new things and keep improving the software and the community. We are not letting that opportunity go either. The Drupal 8 Multilingual Intiative is planning to have another big sprint with three days before and three days after Drupalcon session days (sprinting on both the training day and on the official Drupalcon sprint day of course). We ned people signed up sooner than later to figure out finances, location and tasks for you all. Come to our meetings every other Wednesday in the meantime to figure out what is hot and where can you help out effectively.

Bonus: tips for sprint organizers

This was a huge sprint, and nobody is born to be a good sprint organizer. I've had my bumps as well, and wanted to share some good tips to make your sprint successful, outside of the basic things like getting the right people attend.

  • Don't just focus on building, focus on delivery. You can build lots of stuff at a sprint and then when everybody goes home and business priorities take hold, all that cool stuff will get stale and not get into practice. As the end of sprint gets close, focus more on reviewing what you had and get is committed instead of building a few more new shiny things.
  • Continually engage people, ask them if they are working on something useful, if they are stuck, or need more help. Do a daily standup to have a round of motivation and help people figure out what is going on but also do one on one sync-ups throughout the day. A wasted week is a disaster but a wasted day is still pretty sad.
  • Cross-check your people; teams form around topics, but there are times when certain members would love to go on to other topics where they can be more productive. Don't be stuck with a very long meeting or days of collaboration on an issue with multiple people if not all of them are engaged.
  • Having space for multiple groups of people is great. Separate tables for separate topics help focus work on where things are. Regrouping throughout the day based on the work you do should be encouraged. When people sit with each other, work flows much easier.
  • Get ice cream! Lots. People love ice cream! You cannot even imagine how much they love it. Also, coffee. Almost everybody you ask will say they prefer some kind of coffee, but those will be different kinds. Espresso, Americano, Latte, and so on and on. Don't underestimate the need for coffee.
  • If possible, get sponsors for food and other related expenses. Whether you have someone cooking on the grill for you at night or a selection of menus for lunch at a good place on the street, you get worrying about picking a place, getting there, dividing the bill, and so on off the shoulders of your sprinters. This lets them continue being focused.
  • If you have an open sprint with various people coming, not restricted to your experts only, be prepared with a plan for people coming all the time. Different people have different schedules that they can participate. Have tasks ready for people coming, get them pair up with existing sprinters and review their work (whether code or user interface).

Comments

Great news

Cyclodex's picture

Thanks for this nice summary. Really liked to read this news. (could not participate)
I am still some kind of beginner (about one year of Drupal experience) but using and relaying a lot on i18n in Drupal.

Hope I will be soon/sometime able to help you somehow more than I could now, but I am getting more and more into Drupal.

Thanks for great work, really great Community!

Greetings from Switzerland

Internationalization

Group organizers

Group categories

Group notifications

This group offers an RSS feed. Or subscribe to these personalized, sitewide feeds: