Drupal 7 Usability Study at Google: Results and Notes

You are viewing a wiki page. You are welcome to join the group and then edit it. Be bold!

In february 2012, a usability study of Drupal 7 was done at Google.

…to gain insight into how non-engineering, tech-savvy users interact with the Drupal content management system when they first begin using it.

Videos will be posted soon, watch this space. The testing team will meet again this week to compare notes. Use this page and the comments to add yours!

01.02.2012
Participant 1:
http://lullapad.com/sdjauRi6bE

Participant 2:
http://lullapad.com/OtAN8HzQxn

Participant 3:
http://lullapad.com/BcBsSyNqCu

Participant 4:

03.02.2012
Participant 5:
http://lullapad.com/ZjFfO0IrvX

Participant 6:
http://lullapad.com/3GBR3rmkhA

Participant 7:
http://lullapad.com/K02z64Qpps

Participant 8:
http://lullapad.com/TJnXLmQZte

Comments

Added logs of participants 5

tvn's picture

Added logs of participants 5 and 6.

This is superb! Any plan on

dcmistry's picture

This is superb! Any plan on how we should go about doing the analysis?

Dharmesh Mistry
UX Researcher | Acquia,Inc.

Added a link to this wiki at

xjm's picture

Added a link to this wiki at http://drupal.org/node/1427940 (for now at least; we can remove it if needed when we have the final results).

Raw brain dump

xjm's picture

Incomplete observations I made during the testing (in no particular order). Some of these have issues associated with them already.

  • multi-step creation processes for field UI, views, etc. REALLY confuse people. would a 1 > 2 > 3 navigation be helpful? or should we simply skip advanced settings and let the tester navigate into edit
  • people could not tell how to use menus
  • the navigation menu block confuses people terribly
  • The "add content" link in the navigation block made people think they'd use that to add items to that menu block
  • "tabs" vs. "menus" confused poeple a lot (bartik primary nav theme)
  • previews are terribly confusing
  • teaser preview in particular was unfamiliar / not clear in its relationship to full content
  • links in previews allow people to nuke their content
  • filter tip links allow people to nuke their content
  • the views save button on the upper right confused every participant
  • the seeming distinction between "Add content" and "add new content" confused people
  • On the other hand, many participants were confused by scrolling in the overlay, or thought they should not have to scroll to the bottom to save
  • the local task links in admin menu confused everyone
  • it's important to tag projects with categories on d.o (participant was not able to find date module)
  • the words "image styles" do not have much meaning for people, and it was hard for people to see where to create new ones. people expected something in the context of the display settings tab (once they found it)
  • manage fields was very, very confusing for everyone.
  • people seemed to miss local tasks, or at least took awhile and/or prompting to find them
  • lack of wysiwyg came up right away in the standard profile tests
  • on the other hand, the combination of filter tips + wysiwyg also confused people in the media profile tests--folks thought they could type markup in the wysiwyg, which would actually have been escaped to HTML entities
  • testers had a concept of “developer view” vs. “user view” that corresponded somewhat, but not entirely, to presence/absence of overlay
  • almost all testers did not see filter tips in the first 3-4 looks over the forms, or had to be prompted to see them
  • several testers found admin_menu somewhat overwhelming, and/or spent a lot of time looking through it instead of prompts in the content area
  • testers used toolbar effectively, and generally learned to use “Structure”
  • some testers were confused by changing breadcrumbs and/or expected breadcrumbs to reflect their click history rather than navigation structure
  • testers “lost” their nodes after they were created
  • “add new field” vs “existing field” confused people
  • testers tended to overlook the field label box, and were confused by what to put in the field name box
  • testers’ attention tended to be drawn to vertical tabs. some clicked through all of them; most looked over the content of the first tab. (so, the first tab should contain the most important info?)
  • menu weights confused testers because it was not clear what the weights were relative to. also, one thought the weight selection widget was “crazy” (# of values)
  • testers expected to be able to add fields from the node add form (rather than content type form)
  • most testers managed to add a term reference field eventually (though it took a lot of searching and prompts), but most did not discover the concept of taxonomy/vocabularies/existing terms
  • testers did not really understand the “promoted to front page” concept
  • it was often necessary to prompt testers to “just click save” when they were overwhelmed by a new form
  • testers expected to create a gallery first and then add content to it, but script used the pattern of creating content and then guiding to views to create the content
  • testers all thought views was cool once they understood it a little :)
  • testers had trouble finding manage display tab for field formatters
  • testers often not sure where views, content, etc. “went” when saved--expected putting them in menus, but generally didn’t see how

More usability indeed

ShaneOnABike's picture

This is a really awesome report and I appreciate that you guys have taken time to go through these tests with average users.

I do think that the nomenclature can be misleading or confusing as I have a few clients who really don't understand how to modify and change things in D6 at present (I realize we are talking about D7 but it all applies).

on the other hand, the combination of filter tips + wysiwyg also confused people in the media profile tests--folks thought they could type markup in the wysiwyg, which would actually have been escaped to HTML entities

Totally understandable and a wow for me.

This makes me really want to focus more attention on my development in terms of making it more usable.


Shane Bill
Freelance Web Developer

Clarification Question

technicka's picture

Hi xjm,
Garen and I just went through your notes. Thanks so much for writing this out, it's great to see we are thinking of many of the same issues. I was hoping you could clarify exactly what you mean by "local tasks" like you specified here:

"the local task links in admin menu confused everyone"

Thanks!

sub-tab menu items

xjm's picture

The local tasks are the sub-tabs on a given page, like "view" and "edit" for a node, or "list," "edit," "manage fields" etc. for a taxonomy vocabulary. In the current version of admin_menu these are exposed in the menu structure, but without the context that they would have on a full page.

Here's the admin_menu issue for it:
http://drupal.org/node/946736

I have been watching the

Draven_Caine's picture

I have been watching the posts on these testing, very interesting. Once thing I would like to add is a known issue from almost every user i have talked to. In D6 i created some code to tackle this issue in a custom block. for D7 i created a module to handle it, but in any other words its a massive problem.

How do I create content?
Where is my profile?
(if privatemsg is installed) where are my messages?

Forgot to add the module link

Draven_Caine's picture

Forgot to add the module link http://drupal.org/project/user_details

Added participants 7 and 8

andrewmacpherson's picture

I found them in two different XChat logs, so 7 has unix timestamps and 8 has UTC. (I'm not sure why XChat has logs in two places...)

I'm missing the first couple of minutes of user 8, while I was trying to get XChat to start logging again.

weight selection widget

cpelham's picture

This feedback is great! Being so familiar with Drupal now, I lose sight of what is clearly unintuitive.

One quick thought: rather than raw numerical menu weights, could we have ordinal numbers: 1st, 2nd, 3rd, etc.?

Also, since previews usually do not preview with CSS (at least in some instances) can we warn users of this, add the css, or eliminate the preview? And can we warn users that the preview links are live and will take you away from your unsaved edits?


Christopher Pelham
Director
CRS (Center for Remembering & Sharing)
http://www.crsny.org

CRS (Center for Remembering & Sharing) is an arts & healing center located just south of Union Square in Manhattan.

The one that gets me the most

headdragon's picture

The one that gets me the most on Drupal is the Views so many options that unless you spend hours in there setting parameters and going to look you don't have any use for it.

I have over 200 hours with Drupal and I am still daunted by some functions and some not working and other things. Right now I have comments and poll errors popping up on a new install.

Head Dragon Kid Stevens
Of Web-DrupalDesign .com

Update on study analysis

technicka's picture

Hi everyone,
I just posted an article on my blog about how the analysis has been going so far. Check it out here:
http://technicka.net/tech/digesting-results-drupal-usability-study-4-eme...

If anyone has any more notes along the lines of xjm's post, please share them on this thread so we can compare them with our data.

From that post:

yoroy's picture

The four emerging themes from the study

The issues our users ran into can be categorized by one (or more) of the following problem areas.

  • Interface fails: there are a number of issues that reflect obvious broken elements throughout the Drupal interface. Here we are talking about things like users clicking on links in node preview and then losing all their information, or users continually not seeing the save button from the image styles interface.
  • Terminology confusion: new users do not understand Drupalspeak. We have documented this problem across Drupal as well as the contrib modules we tested. “Why isn’t Views called Lists?” “What is filtered HTML?”
  • Product flow and mental model mismatches: some issues reflect general confusion about where to go next, or how to proceed after taking an action. Users think they are performing one action, yet they are doing something totally different, or they think a function should be found in one place, yet it’s in another. Example: users looking for image styles under manage fields, or the successful installation of a module doesn’t inform a user it still needs to be enabled.
  • Conceptual problems: these are the fun ones. These are the issues when users are just not getting it, and Drupal ain’t doing nothing to help them out. “What is a field? Where does my content live?” etc.

Detailed Notes

GarenCheckley's picture

Yesterday technicka and I posted detailed notes organized by module/category, with links youtube videos, at http://drupal.org/node/1427940.

Usability Report posted

technicka's picture

To view the high-level usability report from this study, see:
http://groups.drupal.org/node/218959

Usability

Group organizers

Group categories

UX topics

Group notifications

This group offers an RSS feed. Or subscribe to these personalized, sitewide feeds: