One of the key things I wanted to do in the Drupal 8 Multilingual Initiative after the December 1st feature freeze hits is to go and do lots of user testing of the functionalities we built and adapt the user interfaces and flows to make it easier to build multilingual sites (and then test it again). Not many people believed this would actually happen (when you say in an issue you'll follow up with more user testing, and so on, it can be seen as just a way out to move on). However I've got outstanding help from Bojhan Somers, Dharmesh Mistry and Lisa Rex on best practices for user testing. I not only got help on the tools to use, but also building out a concrete test plan with Dharmesh and Bojhan actively involved. To say the test was well prepared is an understatement.
Timing and focus of the study
Although it was a first for me to do testing, I've had an extraordinary opportunity on the second (community) day of DrupalCamp Budapest on November 25th to do user testing on multiple participants, so I grabbed the opportunity. The three key questions we wanted to get answers on were the following:
- Can users add a foreign language to an existing site?
- Can users configure translatability / language options for entities?
- Can users do a basic node translation?
This might not look very ambitious, but it let us test the base functionality we are building and still find plenty of usability issues (and plain code bugs).
The tests were conducted with 5 people total. Three of them were testin Drupal 8 with me and 2 of them with István Palócz. We used Google Hangouts with the "on air" option to stream to Youtube right away and retain the recording for note-taking later. All the test conductors and participants were native Hungarian speakers, so all the videos are fully in Hungarian. We did not record the first participant (Éva Rauscher), due to computer resources lacking, but the other four were recorded: István Palócz himself, Tamás Pintér, Attila Laskay and István Mazál I took notes with the first participant as much as possible.
The participants were from all over the spectrum, one of them never used Drupal before, we had an absolute expert who even teaches Drupal, one developer and two Drupal site builders with different levels of experience (one built Drupal sites before but not multilingual sites). So the feedback points accumulated are coming from all angles.
Task 1: adding Hungarian to an English site
In this task, what we have been looking for is that people realize that language setup is provided by a module that needs to be enabled, understand how to add a language and explain the three special languages in core that were added in Drupal 8 (not applicable, not specified and multiple).
Both experienced and non-experienced users had issues figuring out that language needs a module. Every single participant looked at the Configuration screen first under Regional and languages and looked at the Regional settings screen (which lets you set up timezones and country but no language). Most stopped here and wondered how to add language support.
On the way we tested the toolbar, the new module list and the configuration interface as well:
- Our beginner said: "Extend: is this a wall plug?!", "I did not know what Extend means", "I did not know what the wall plug meant", "Is still don’t know what it means", "It meant 'connect' to something to me, maybe a plus sign there would help"
- People experienced with Drupal were puzzled the Modules list changed to Extend (eg. "I did not believe I’d need to look in Extend for modules"), not all of them were immediately comfortable with it, but said they can get used to it
- Two participants enabled all modules which looked related to languages (Language, Locale, Entity translation). Our beginner had these comments: "I don’t know what it means exactly: 'Entity translation'"; "Language module description too long, hard to understand", "Oh, I also need locale module?"; "Would be good to know if we need it"
- Participants who only enabled Language (and Locale) modules were mostly puzzled as to why they have no options to translate things; did not understand before some futzing around they'd need another module.
- Our beginner read each module description and noted the Language module description was way long. Opening down the description (because it did not fit the length of the screen), exclaimed: "Now what is wrong with this?" seeing the red "disabled" notices along modules dependent on Language).
There were no problems observed with adding Hungarian once the Language module was enabled: "The blue button is very prominent, looks good.", "Looked good after I enabled the language module."
None of the participants noticed the message showing up on the top of the page guiding them to enable the language switcher block once they had 2 languages. Nobody noticed it being there.
We also queried them about the usefulness of the special languages "not specified", "not applicable" and "multiple". This was probably the component that tested the worst of all. One participant understood "multiple" to let him set up content to be translatable and was attached to this idea for an extended time on various UIs. Choice quotes:
- "I see some strange special languages here"; "I don’t understand why would there be a "not specified" language"; "These are not languages, these are settings", "I don’t understand what these are doing here; these should show up where I actually set a language on something", "I don’t know why can I make it a default; oh, I cannot make it a default; the only thing I can do is to reorder them; I don’t understand the usefulness of that"; "Should I be able to turn on more help to let me know what are these options?"; "It would be much easier if there would be more help on these options, or moved to settings"
- "Not clear what Multiple languages mean; is this related to one content item or field?"; "Should I use this if I want multilingual content? If I want to say a content item is Hungarian or English?", "I don’t fully understand the difference between not specified and not applicable language", "I think I’d set this to Multiple so then it would become translatable" (on content type settings page); "So I think maybe I need to go and set 'Multiple' at the [main] configuration screen too to make it translatable"; "So I assume if I set it to Multiple, it will be translatable, but then .... where would I set the actual language?!?"; "If I set this to 'Multiple', then I expected a field to show up to specify the specific language"; "So I think I’d search for translatability in the configuration; I’d set Multiple as the default language; but I cannot check the radio button there, it does not work"
- "I don’t fully understand the difference between not specified and not applicable language"
In short, looks like "Multiple" should probably go away and the other two would need to go to an advanced setting collapsed fieldset(?). Definitely needs more discussion.
There were problems with the language switcher blocks as well. People were definitely puzzled there is a user interface and a content language switcher. Choice quotes:
- "I need a block that switches the language **but always**, I don’t get what is the 'content' and 'user interface text' would do differently"
- "We definitely need a block that changes both languages at once [content, interface]. So looks like I need to enable both blocks? The visitors would be interested in content, so I’d enable that for them."; "At least title the two blocks differently if we have two of them; my next step would be to rename it to Language for content and ..."
(Most people did not go to detection and selection to experience the confusion of the duplicated tables there).
Task 2: making things translatable (across entity types)
The cornerstone of this task was the patch from http://drupal.org/node/1810386 which we applied to each test environment.
Where this tested worst is when entity translation was not enabled and the "Multiple" language item did drive the test participant to believe he already has this functionality. For experienced site builders, they did not expect the centralized UI, they went individually to configure per entity (node, user, etc.). Some feedback from those:
- "I’d love a cross-link from people to account settings, I only this because I’ve been working with Drupal for years"; "It is disturbing that 'people' settings are called 'account' settings and accounts are 'users'"
- When user attempted to go to their profile after enabling translation: "So now I go to my own account; did I go there?" (nothing happens when clciking on account name in toolbar while overlay is open) "This does not actually lead me there" (toolbar second line opens, does not load any user page)
None of the participant noticed the guiding message text appearing right after the entity translation module was enabled. One participant noted right after he clicked away "I should have read that text". That was it. It had no practical usefulness. The module name was also confusing: "I don’ think Entity translation would say anything to the average user".
Less experienced people found the central settings screen pretty easily, more experienced people took some time to figure out there would be a central screen (they did not expect it to exist). Once people found it regardless of experience, they loved it:
- "So that it set it up to all [the fields] be translatable for me is superb"; "I think this was very easy"; "It looks like well made; looks good"
- "It is a fantastic opportunity that I can configure everything at once UI, it looks like a killer feature"
- "Wow, this looks cool"
- "Ahah, content translation settings looks like a good place to configure everything at once"
- "This looks good that I can do it all on one page"
People were confused by the "hide language selector" checkbox. Some misunderstood whether to check it or not to hide it (they assumed it works like if labeled "show language selector"). Most people took a little time to understand the usefulness of the functionality but then got it.
- "Why would it be good to hide the language selector?"
- "What is going to happen if I enable translation but hide the language selector?"; "Looks like a great feature to create node in current interface language and hide the selector. Looks useful." (But then **unchecked** hide language selector to *hide* the language selector); shortly after: "I wonder why the language selector is showing up, probably because I am an admin"
- "Interesting that if I enable translation, why is it possible to hide the language selector; I assume this will be a problem for many"; "It would make sense to hide the language selector if there is only one language; would avoid Drupal offering more than needed"
People were confused as to why certain things would be useful to translate but then got it after a while. And even cited use cases in some cases:
- "I don’t see a reason to ever translate comments; I got confused by the content type names under comments; thought I am setting up content types"
- "What does it mean that I translate a user?"; "Ah, so if a user has bio or something, that makes sense to be translatable; but I never needed that"; "I’ve had a client before needing comment translation; comments on photos for a professional photographer where the comments were in English, the client said this looks lame that the comment was not on the Hungarian page; so they hired translators to cross-translate the comments"
- "It might make sense to have nodes before comments, since comments come from nodes"
There were some interaction issues/bugs noticed with the page.
- One participant noted that tables show up outside the viewport when the checkbox is checked and that is hard to notice in general. "I’d love if I check comments the action would happen close to the checkbox; if I click users, the table shows up outside of my screen"
- "If I turn on/off the translatability for type, it overrides previous settings (that I already set elsewhere)"
- The relation of checkboxes fields/types was confusing for 2 parcitipants: "I’m confused that the basic page’s checkbox does not belong to articles, but it it in the same sequence"; "Just seeing that the fields are individually configured under the content types, although the field name is indented, I did not realize it is a field not a content type"
- "What would happen if the article is translatable but none of its fields? This sounds confusing"
- "It would be great if it would not throw an error if none of the [bundle] types were selected, like a tableselect, if I uncheck all, it unchecks the top checkbox"
There was even a suggestion from a participant if there are many node types / entity types: "I miss a functionality that if I have 15 content types, I could copy-paste the language settings to all of them“ (default language, exposing of language selector)".
The types of fields that should be marked translatable right away also got some coverage. One participant noted imagefields should not be translatable by default. Another one was thinking of a use case where it would make sense, so he enabled it. The other three did not think about the specifics of this.
Task 3: actual translation
People easily found the translate tab on nodes.
The main bug that was encountered is that the existing Body field content was not carried over from before the content type was configured translatable. People were very confused by the Body disappearing. Given this is simply a bug, I'll not cite quotes on this. When we figured out the node render falls back to Hungarian body for English as well since it does not have an English body, it was confusing. Participants said "I’m missing settings for fallback behavior", "I’m missing settings as to how a node would display depending on the page’s language".
We did not test going back to edit the translation, one participant confused the Edit tab for editing translations with the Translations tab for a second. We should test this more.
People also found it disconcerting that the title was not translatable. They remembered they did not configure title to be translatable, but it was obviously not an option: "Oh, so I did not set up the title to be language-variant"; "I cannot translate the title? I’m going to cry"; "I want to make title translatable, this worked before [in earlier versions], but not now".
Every single participant opened down the "source language" collapsed fieldset on top of the translation submission form and looked at it. One participant noted "It looks interesting that I can change the source language, cool functionality", others ignored it. Also (almost?) every participant went into the Translation vertical tab on the translation submission screen thinking it would be important which only has one checkbox to mark translations outdated. Most of them did not fully grok what it is useful for and considered it very prominent vs. the usefulness when creating translations.
We definitely have lots of things we can improve on. The above findings can be used to open issues and provide supporting arguments for changes proposed. I could have pointed to specific points in videos, but given it is all in Hungarian, it did not seem very useful to do, so did not make specific notes on that level.
I'd love if we could work on solving several issues found and re-test the user interface with another group of participants. I'm looking for people to help open / fix issues as well as to conduct user testing either on the existing system and/or once the improvements are in. Contact me if you want to be involved!