First Drupal 8 multilingual user testing results

We encourage users to post events happening in the community to the community events group on https://www.drupal.org.
Gábor Hojtsy's picture

One of the key things I wanted to do in the Drupal 8 Multilingual Initiative after the December 1st feature freeze hits is to go and do lots of user testing of the functionalities we built and adapt the user interfaces and flows to make it easier to build multilingual sites (and then test it again). Not many people believed this would actually happen (when you say in an issue you'll follow up with more user testing, and so on, it can be seen as just a way out to move on). However I've got outstanding help from Bojhan Somers, Dharmesh Mistry and Lisa Rex on best practices for user testing. I not only got help on the tools to use, but also building out a concrete test plan with Dharmesh and Bojhan actively involved. To say the test was well prepared is an understatement.

Timing and focus of the study

Although it was a first for me to do testing, I've had an extraordinary opportunity on the second (community) day of DrupalCamp Budapest on November 25th to do user testing on multiple participants, so I grabbed the opportunity. The three key questions we wanted to get answers on were the following:

  • Can users add a foreign language to an existing site?
  • Can users configure translatability / language options for entities?
  • Can users do a basic node translation?

This might not look very ambitious, but it let us test the base functionality we are building and still find plenty of usability issues (and plain code bugs).

Participants

The tests were conducted with 5 people total. Three of them were testin Drupal 8 with me and 2 of them with István Palócz. We used Google Hangouts with the "on air" option to stream to Youtube right away and retain the recording for note-taking later. All the test conductors and participants were native Hungarian speakers, so all the videos are fully in Hungarian. We did not record the first participant (Éva Rauscher), due to computer resources lacking, but the other four were recorded: István Palócz himself, Tamás Pintér, Attila Laskay and István Mazál I took notes with the first participant as much as possible.

The participants were from all over the spectrum, one of them never used Drupal before, we had an absolute expert who even teaches Drupal, one developer and two Drupal site builders with different levels of experience (one built Drupal sites before but not multilingual sites). So the feedback points accumulated are coming from all angles.

Task 1: adding Hungarian to an English site

In this task, what we have been looking for is that people realize that language setup is provided by a module that needs to be enabled, understand how to add a language and explain the three special languages in core that were added in Drupal 8 (not applicable, not specified and multiple).

Both experienced and non-experienced users had issues figuring out that language needs a module. Every single participant looked at the Configuration screen first under Regional and languages and looked at the Regional settings screen (which lets you set up timezones and country but no language). Most stopped here and wondered how to add language support.

On the way we tested the toolbar, the new module list and the configuration interface as well:

  • Our beginner said: "Extend: is this a wall plug?!", "I did not know what Extend means", "I did not know what the wall plug meant", "Is still don’t know what it means", "It meant 'connect' to something to me, maybe a plus sign there would help"
  • People experienced with Drupal were puzzled the Modules list changed to Extend (eg. "I did not believe I’d need to look in Extend for modules"), not all of them were immediately comfortable with it, but said they can get used to it
  • Two participants enabled all modules which looked related to languages (Language, Locale, Entity translation). Our beginner had these comments: "I don’t know what it means exactly: 'Entity translation'"; "Language module description too long, hard to understand", "Oh, I also need locale module?"; "Would be good to know if we need it"
  • Participants who only enabled Language (and Locale) modules were mostly puzzled as to why they have no options to translate things; did not understand before some futzing around they'd need another module.
  • Our beginner read each module description and noted the Language module description was way long. Opening down the description (because it did not fit the length of the screen), exclaimed: "Now what is wrong with this?" seeing the red "disabled" notices along modules dependent on Language).

There were no problems observed with adding Hungarian once the Language module was enabled: "The blue button is very prominent, looks good.", "Looked good after I enabled the language module."

None of the participants noticed the message showing up on the top of the page guiding them to enable the language switcher block once they had 2 languages. Nobody noticed it being there.

We also queried them about the usefulness of the special languages "not specified", "not applicable" and "multiple". This was probably the component that tested the worst of all. One participant understood "multiple" to let him set up content to be translatable and was attached to this idea for an extended time on various UIs. Choice quotes:

  • "I see some strange special languages here"; "I don’t understand why would there be a "not specified" language"; "These are not languages, these are settings", "I don’t understand what these are doing here; these should show up where I actually set a language on something", "I don’t know why can I make it a default; oh, I cannot make it a default; the only thing I can do is to reorder them; I don’t understand the usefulness of that"; "Should I be able to turn on more help to let me know what are these options?"; "It would be much easier if there would be more help on these options, or moved to settings"
  • "Not clear what Multiple languages mean; is this related to one content item or field?"; "Should I use this if I want multilingual content? If I want to say a content item is Hungarian or English?", "I don’t fully understand the difference between not specified and not applicable language", "I think I’d set this to Multiple so then it would become translatable" (on content type settings page); "So I think maybe I need to go and set 'Multiple' at the [main] configuration screen too to make it translatable"; "So I assume if I set it to Multiple, it will be translatable, but then .... where would I set the actual language?!?"; "If I set this to 'Multiple', then I expected a field to show up to specify the specific language"; "So I think I’d search for translatability in the configuration; I’d set Multiple as the default language; but I cannot check the radio button there, it does not work"
  • "I don’t fully understand the difference between not specified and not applicable language"

In short, looks like "Multiple" should probably go away and the other two would need to go to an advanced setting collapsed fieldset(?). Definitely needs more discussion.

There were problems with the language switcher blocks as well. People were definitely puzzled there is a user interface and a content language switcher. Choice quotes:

  • "I need a block that switches the language **but always**, I don’t get what is the 'content' and 'user interface text' would do differently"
  • "We definitely need a block that changes both languages at once [content, interface]. So looks like I need to enable both blocks? The visitors would be interested in content, so I’d enable that for them."; "At least title the two blocks differently if we have two of them; my next step would be to rename it to Language for content and ..."

(Most people did not go to detection and selection to experience the confusion of the duplicated tables there).

Task 2: making things translatable (across entity types)

The cornerstone of this task was the patch from http://drupal.org/node/1810386 which we applied to each test environment.

Where this tested worst is when entity translation was not enabled and the "Multiple" language item did drive the test participant to believe he already has this functionality. For experienced site builders, they did not expect the centralized UI, they went individually to configure per entity (node, user, etc.). Some feedback from those:

  • "I’d love a cross-link from people to account settings, I only this because I’ve been working with Drupal for years"; "It is disturbing that 'people' settings are called 'account' settings and accounts are 'users'"
  • When user attempted to go to their profile after enabling translation: "So now I go to my own account; did I go there?" (nothing happens when clciking on account name in toolbar while overlay is open) "This does not actually lead me there" (toolbar second line opens, does not load any user page)

None of the participant noticed the guiding message text appearing right after the entity translation module was enabled. One participant noted right after he clicked away "I should have read that text". That was it. It had no practical usefulness. The module name was also confusing: "I don’ think Entity translation would say anything to the average user".

Less experienced people found the central settings screen pretty easily, more experienced people took some time to figure out there would be a central screen (they did not expect it to exist). Once people found it regardless of experience, they loved it:

  • "So that it set it up to all [the fields] be translatable for me is superb"; "I think this was very easy"; "It looks like well made; looks good"
  • "It is a fantastic opportunity that I can configure everything at once UI, it looks like a killer feature"
  • "Wow, this looks cool"
  • "Ahah, content translation settings looks like a good place to configure everything at once"
  • "This looks good that I can do it all on one page"

People were confused by the "hide language selector" checkbox. Some misunderstood whether to check it or not to hide it (they assumed it works like if labeled "show language selector"). Most people took a little time to understand the usefulness of the functionality but then got it.

  • "Why would it be good to hide the language selector?"
  • "What is going to happen if I enable translation but hide the language selector?"; "Looks like a great feature to create node in current interface language and hide the selector. Looks useful." (But then **unchecked** hide language selector to *hide* the language selector); shortly after: "I wonder why the language selector is showing up, probably because I am an admin"
  • "Interesting that if I enable translation, why is it possible to hide the language selector; I assume this will be a problem for many"; "It would make sense to hide the language selector if there is only one language; would avoid Drupal offering more than needed"

People were confused as to why certain things would be useful to translate but then got it after a while. And even cited use cases in some cases:

  • "I don’t see a reason to ever translate comments; I got confused by the content type names under comments; thought I am setting up content types"
  • "What does it mean that I translate a user?"; "Ah, so if a user has bio or something, that makes sense to be translatable; but I never needed that"; "I’ve had a client before needing comment translation; comments on photos for a professional photographer where the comments were in English, the client said this looks lame that the comment was not on the Hungarian page; so they hired translators to cross-translate the comments"
  • "It might make sense to have nodes before comments, since comments come from nodes"

There were some interaction issues/bugs noticed with the page.

  • One participant noted that tables show up outside the viewport when the checkbox is checked and that is hard to notice in general. "I’d love if I check comments the action would happen close to the checkbox; if I click users, the table shows up outside of my screen"
  • "If I turn on/off the translatability for type, it overrides previous settings (that I already set elsewhere)"
  • The relation of checkboxes fields/types was confusing for 2 parcitipants: "I’m confused that the basic page’s checkbox does not belong to articles, but it it in the same sequence"; "Just seeing that the fields are individually configured under the content types, although the field name is indented, I did not realize it is a field not a content type"
  • "What would happen if the article is translatable but none of its fields? This sounds confusing"
  • "It would be great if it would not throw an error if none of the [bundle] types were selected, like a tableselect, if I uncheck all, it unchecks the top checkbox"

There was even a suggestion from a participant if there are many node types / entity types: "I miss a functionality that if I have 15 content types, I could copy-paste the language settings to all of them“ (default language, exposing of language selector)".

The types of fields that should be marked translatable right away also got some coverage. One participant noted imagefields should not be translatable by default. Another one was thinking of a use case where it would make sense, so he enabled it. The other three did not think about the specifics of this.

Task 3: actual translation

People easily found the translate tab on nodes.

The main bug that was encountered is that the existing Body field content was not carried over from before the content type was configured translatable. People were very confused by the Body disappearing. Given this is simply a bug, I'll not cite quotes on this. When we figured out the node render falls back to Hungarian body for English as well since it does not have an English body, it was confusing. Participants said "I’m missing settings for fallback behavior", "I’m missing settings as to how a node would display depending on the page’s language".

We did not test going back to edit the translation, one participant confused the Edit tab for editing translations with the Translations tab for a second. We should test this more.

People also found it disconcerting that the title was not translatable. They remembered they did not configure title to be translatable, but it was obviously not an option: "Oh, so I did not set up the title to be language-variant"; "I cannot translate the title? I’m going to cry"; "I want to make title translatable, this worked before [in earlier versions], but not now".

Every single participant opened down the "source language" collapsed fieldset on top of the translation submission form and looked at it. One participant noted "It looks interesting that I can change the source language, cool functionality", others ignored it. Also (almost?) every participant went into the Translation vertical tab on the translation submission screen thinking it would be important which only has one checkbox to mark translations outdated. Most of them did not fully grok what it is useful for and considered it very prominent vs. the usefulness when creating translations.

Summary

We definitely have lots of things we can improve on. The above findings can be used to open issues and provide supporting arguments for changes proposed. I could have pointed to specific points in videos, but given it is all in Hungarian, it did not seem very useful to do, so did not make specific notes on that level.

I'd love if we could work on solving several issues found and re-test the user interface with another group of participants. I'm looking for people to help open / fix issues as well as to conduct user testing either on the existing system and/or once the improvements are in. Contact me if you want to be involved!

Comments

Thank you!

dcmistry's picture

Gabor and team --

This is pretty rad that you guys did the study. The results do serve good pointers as to what the ux pain points are.

Kudos!

Dharmesh Mistry
UX Researcher | Acquia,Inc.

yeah

eigentor's picture

fantastic work.
This must be so painful every time when people are just normal and have a hard time with the interface...

Life is a journey, not a destination

This is fabulous! IMO

dalin's picture

This is fabulous! IMO Drupal's shortcoming in i18n has not been a lack of features, but lack of usability.

--


Dave Hansen-Lange
Director of Technical Strategy, Advomatic.com
Pronouns: he/him/his

Interesting

plach's picture

This is really useful data to work on. Here is my personal list of solutions for the outlined issues.

Disclaimer: I'm no UX expert thus bear with me and my feedback, please ;)

Every single participant looked at the Configuration screen first under Regional and languages and looked at the Regional settings screen (which lets you set up timezones and country but no language)

Since the language concept is baked deeply into core we could have a stub Language menu entry provided by the system module, which would only provide a small help text including a (CSRF-safe) link clicking which would enable the Language module. The stub menu entry would be overridden by the Language module. We might have links to enable translation modules too.

About the 'extend' label: I was doubtful about it too, the first time I saw it. Maybe it's just me but I think "Add-ons" would work better here, if we want to avoid the module terminology. As a bonus it would be a name, which would be consistent with the rest of the same-level toolbar items.

None of the participants noticed the message showing up on the top of the page guiding them to enable the language switcher block once they had 2 languages. Nobody noticed it being there.

I don't think this is a big deal, I bet many other blocks suffer from the very same "visibility" problem. I'd be tempted to classify this as part of the learning process.

We also queried them about the usefulness of the special languages "not specified", "not applicable" and "multiple". This was probably the component that tested the worst of all.

I don't see a big reason to have those in the language overview page: if I'm not mistaken you can only reorder them, not a killer feature IMHO :)
It seems the drawbacks vastly outweight the benefits here...

There were problems with the language switcher blocks as well. People were definitely puzzled there is a user interface and a content language switcher.

This should be solved by the language dectection page overhauling: by default only UI language would be configuable again, and this would leave only one langauge switcher block available.

None of the participant noticed the guiding message text appearing right after the entity translation module was enabled. One participant noted right after he clicked away "I should have read that text". That was it. It had no practical usefulness.

We could exploit modal dialogs to have popup messages, which should be fairly less easy to ignore :)

"If I turn on/off the translatability for type, it overrides previous settings (that I already set elsewhere)"

This is by design: if you turn off translation, well, you turn off translation :)

The main bug that was encountered is that the existing Body field content was not carried over from before the content type was configured translatable.

This is a known issue: we are not doing bulk migrations for all fields, since when Field API is fully converted to the ENtity Field API, we won't need migrations anymore.

When we figured out the node render falls back to Hungarian body for English as well since it does not have an English body, it was confusing. Participants said "I’m missing settings for fallback behavior", "I’m missing settings as to how a node would display depending on the page’s language".

In ET-D7 we actually have a setting to disable language fallback, however node translation works exactly this way: if you don't have an English node you see the Hungarian node even if your current languge is English.

People also found it disconcerting that the title was not translatable.

Not hard to believe :) It will be translatable by default, obviously.

Also (almost?) every participant went into the Translation vertical tab on the translation submission screen thinking it would be important which only has one checkbox to mark translations outdated. Most of them did not fully grok what it is useful for and considered it very prominent vs. the usefulness when creating translations.

http://drupal.org/node/1807800 is going to add more options there, although not for nodes.

Are you still thinking: We

YesCT's picture

Are you still thinking:

We could exploit modal dialogs to have popup messages, which should be fairly less easy to ignore :)

If so, we'll need an issue for it.

Cathy Theys

very agressive

Gábor Hojtsy's picture

I don't think we should ever go there, sounds very agressive.

Looks like this needs help

yoroy's picture

Looks like this needs help with opening the right issues in the core queue for improving around the problems found? Tag them with 'usability' and invent a tag to label it as an issue found during this particular test, 'budapest2012' or something.

Great to see this being done. Smart timing too, still time to improve things before release. Excellent job all!

Thinking out loud the kind of

Bojhan's picture

Thinking out loud the kind of issues this creates;

"Two participants enabled all modules which looked related to languages (Language, Locale, Entity translation). Our beginner had these comments: "I don’t know what it means exactly: 'Entity translation'"; "Language module description too long, hard to understand", "Oh, I also need locale module?"; "Would be good to know if we need it"

Participants who only enabled Language (and Locale) modules were mostly puzzled as to why they have no options to translate things; did not understand before some futzing around they'd need another module."

  1. This is confusing, could we potentionally rename some of them? Possibly group them togheter, frankly I never even know what Locale is for :) Sounds like this is a serious problem if not directed, this can be an issue.

"None of the participants noticed the message showing up on the top of the page guiding them to enable the language switcher block once they had 2 languages. Nobody noticed it being there."

  1. We poluted the messages area too much, this can possibly be solved by the "new blocks" concept.

"I see some strange special languages here"; "I don’t understand why would there be a "not specified" language"; "These are not languages, these are settings", "I don’t understand what these are doing here; these should show up where I actually set a language on something", "I don’t know why can I make it a default; oh, I cannot make it a default; the only thing I can do is to reorder them; I don’t understand the usefulness of that"; "Should I be able to turn on more help to let me know what are these options?"; "It would be much easier if there would be more help on these options, or moved to settings"

  1. This needs a seperate issue, I think its clear merging this into the current list is simply not usable. I guess we might want to look into the idea of making this functionality opt-in rather, than enabled by default?

"I don’t think Entity translation would say anything to the average user".

  1. I know we settled on Entity translation, but it seems that it doesn't really direct users. Can't it be like "Site" or "Content" translation?

  2. Very happy to see the actual multinlingual entity setup page, tested so well :) We should figure out over a i18n or ux meetup how to split up the issues identified.

"Every single participant opened down the "source language" collapsed fieldset on top of the translation submission form and looked at it. One participant noted "It looks interesting that I can change the source language, cool functionality", others ignored it. Also (almost?) every participant went into the Translation vertical tab on the translation submission screen thinking it would be important which only has one checkbox to mark translations outdated. Most of them did not fully grok what it is useful for and considered it very prominent vs. the usefulness when creating translations."

  1. Are there other settings in here?

Does the "new blocks" concept

YesCT's picture

Does the "new blocks" concept have an issue?

Cathy Theys

two issues opened

Gábor Hojtsy's picture

We have two issues for two starter issues, more to come:

one more

Gábor Hojtsy's picture

One more for today:

Issue opened regarding language switcher blocks

YesCT's picture

re-name to differentiate content vs UI language switcher blocks
http://drupal.org/node/1874102

Cathy Theys

Identified in task

YesCT's picture

Identified in task 1:
http://drupal.org/node/1891096 issue for confusion around the plug icon for extending/modules
http://drupal.org/node/1833184 already fixed in part renaming modules, also shortening the descriptions
http://drupal.org/node/1891126 red disabled word mistaken for an indicator something is wrong with the module participant wanted to enable

Cathy Theys

More issues from task

YesCT's picture

More issues from task 1:
http://drupal.org/node/1891450 message to enable language switcher block not noticed
http://drupal.org/node/1833022 (was already opened) about the detection and selection confusion we assume will happen

Cathy Theys

From Task 2: already

YesCT's picture

From Task 2:
already existing: http://drupal.org/node/715972#comment-6951336 people and config accounts sections need to be linked

others points about entity/content translation have already been addressed

Cathy Theys

Task

YesCT's picture

Task 3:
http://drupal.org/node/1498674 make title translatable. (work on this to help: http://drupal.org/node/1818556 it's blocking)

some other things already solved like
http://drupal.org/node/1869328 (restore simplicity: removes system languages from table)

Cathy Theys

Please check (here and

YesCT's picture

Please check (here and http://drupal.org/node/1868058) if any follow-ups were missed.

Cathy Theys

looks good, thanks!

Gábor Hojtsy's picture

Looks good, thanks for opening the followups!

Usability

Group organizers

Group categories

UX topics

Group notifications

This group offers an RSS feed. Or subscribe to these personalized, sitewide feeds: