Usability Testing Plan

IMPORTANT: Editing this Document and Collaboration

This wiki page was created from node/7878 so that it can be collaborated on. This copy of the document is the primary/trunk/head copy of the document where all changes should be merged. Discuss this document on node/7878. Bevan/

Plan for Usability Testing at UMN, February 2008

Context

The University of Minnesota Libraries has made their usability lab available for formal usability testing on Drupal in February. For more details see the official announcement and follow-up discussion in the usability group.

Overview

This will be the first time Drupal has undergone any formal usability testing. The objective of this document is to identify audiences, scenarios and tasks that communicates a baseline understanding of Drupal’s usability for common tasks. Overall testing priority will be given to a user role we are describing as that of the Site Maintainer.

High Level Usability Goals

This round of testing will focus on the needs of users who maintain departmental or corporate websites and will seek to identify usability issues within the Drupal CMS that are associated with these needs.

Specific Goals Might Include:
* @COMMENT These are provisional goals; please feel free to add new goals or new versions of these goals /libsys

• Users with minimal web content management technology experience should be able to perform basic content management tasks with minimal direction. Such tasks might include adding pages; assigning links to content; locating and editing content; reverting previous revisions of content; adding comments; or assigning pages to "site sections".
• Users with an intermediate to advanced level of experience with web content management technologies should be able to perform critical site building and maintenance tasks with modest direction. Critical tasks might include creating new content data entry forms (content types); establishing permissions for interaction with site features; performing high-level site settings tasks, such as configuring the look and feel of the site, enabling/disabling site features (module settings); or establishing "site sections" and menus of links (outline, menus, blocks).

Open Issues

1. We have yet to test these scenarios and do not have a sense for time to complete them. Evaluators (testers) will have about 1.5 hours each.
   • I think it's one hour plus debriefing. B/
   • Very good idea to run someone through the scenario to see if time is sufficient. Normally it never is, so one has to cut down...

Details

• We will have 8 users for the entire testing program. This is approximately an industry standard number.
• Each evaluator will have about 1 hours to complete the alloted tasks. There will be another 30 minutes for debriefing and preparing for the next test.
• We have 4 evaluators testing for each of the Content Contributor and Site Maintainer target audiences.
  • We decided (in the Jan 14th Kickoff Meeting) to devote all 8 slots to the Site Maintainer Role for two primary reasons (1) you really do need to devote all 8 slots to a given set of tasks to find most problems w/in those tasks and (2) the Site Maintainer UI is the more testable of the two as Content Contributors often see a radically reduced version of Drupal as a result of theming. Admins often do this so as to make contributor tasks as simple as possible. /libsys
  • This wiki will be update soon to reflect these changes and others decided w/in the 14th meeting. /libsys Done. /codyh

Key dates

14 Jan – Kickoff Meeting

A first pass review of tasks, audiences, this document, etc.

14 Feb– Scenario/Prototype Review

In depth look at scenarios and prototype walk through.

26 and 27 Feb – Testing

Eight evaluators will test the product in the lab.

28 Feb – Issues Analysis

A review of testing outcomes.

3 - 6 Mar - Presentation

Presentation of results to the drupal community at DrupalCon Boston

Attendees of On-Site Testing

• Dries Buytaert
  • acquia
  • http://buytaert.net
• Angela “webchick” Byron
  • lullabot
• Nathaniel “catch” Catchpole
  • http://libcom.org
• Neil “drumm” Drumm
  • http://www.advomatic.com
• Greg “greggles” Knaddison
  • http://knaddison.com
• Kieran “Amazon” Lal
  • CivicSpace labs
  • Acquia
• Karen “KarenS” Stevenson
  • http://karen-stevenson.com
• Bevan "Bevan" Rudge
  • CivicActions
  • http://drupal.geek.nz

University of Minnesota Contacts

• Chad “libsys” Fennell
  • fenne035@umn.edu
• Cody “codyh” Hanson
  • hans1794@umn.edu

Audience(s)

Site Maintainer (editor, or sub-admin)

• authenticated user with access to manage user permissions and content, including blocks, taxonomy, menus, comments, and nodes.

The following user scenarios have been declared out of scope for this first round of testing. If you are interested in testing these users, please take a look at the work we've done thus far here.

Visitor

• anonymous user with permission to read, navigate, comment (with moderation), register
• once registered, authenticated user with permission to create content, comment (without moderation), view users
  • @comment: This currently isn't catered for in the plan (this document), but is under consideration. B/

Content Contributor

• authenticated user with permission to create content such as pages, menu items, and comments.

Admin

• authenticated user with access to all site settings, responsible for installation and configuration.
  • @comment: It looks like we're leaving this target audience out. B/
  • @comment: We need to determine high value admin tasks. What sorts of top-level config stuff does the community really want tested (Setting themes, site information, etc)? /libsys

Site Maintainer Scenario

Open Issues

• This seems like a lot for one hour, although is an excellent set of tasks. B/

Summary

The goal of these tests is to create a scenario that reasonably approximates the common tasks performed by a site maintainer. These tasks will require some familiarity with web content management systems in general, if not Drupal specifically. We will endeavor to capture testers’ initial impressions of various aspects of Drupal, and then follow up with such instruction as is necessary to complete the tasks.

Tasks will include enabling modules, setting up new users, adding content blocks, establishing menus and vocabulary configuration. Possible secondary tasks may include theme and site information configuration.

An ideal, or most efficient, path for each task is listed in italics following each task.
@COMMENT Note that I've added these ideal paths in on 2/7. The second task's ideal path is currently incomplete. I'll have a final version shortly. /codyh

Tasks

@COMMENT: Sooooo, I'm doing some housecleaning here. I'm pasting in revised versions of these tasks. They retain the fundamentals, but recast the narrative in accordance with some guidance we've gotten from the usability lab. I'm retaining the previous versions (struck through below) to keep a record of our discussion. /codyh

You are the webmaster for a campus library that has chosen a new content management system to manage part of its website.

You have been asked to configure the system so that your librarians can log in to create web pages for workshops they teach at the library. These workshop pages should include the workshop name, description, and an indication of whether the workshop is held during the day or in the evening. In order to make it as easy as possible for your librarians, you must configure the site such that all they need to do is to enter the required information about their workshop in a form.

As webmaster, you also want to be able to classify these workshop pages and other pages on the site according to their relevance to students in specific academic departments.

These pages should be listed on the left-hand side of your site under the heading “Workshops”.

1. Build a form to allow users to create workshop pages. Each page should contain the following pieces of information: workshop name, description, and whether the workshop is held during the day or in the evening.
   IDEAL PATH: LH Menu > Content management > Content types > Add content type > Name: "Workshop" or like Type: "workshop " or like > Save content type > Content types > edit "Workshop" content type > Add field > Click "enable one" link in warning message > Enable "Text" module under CCK > Save > LH Menu > Content management > Content types > Edit Course Page content type > Add field >Field name "workshop_name" or like > Field type "Integer/Text Field" or like > Create field > Add field > Field name "description" or like > Field type "Integer/Text Field" or like > Create field > Add field > Field name "time" or like > Field type "Integer/Text Field" or like > Create field

2. You want to enable one of your librarians, Bob Bruininks, to log in and create a workshop page. Bob should have the same level of access to the site as all other librarians, but should not be able to control other librarians’ access or accounts.
   IDEAL PATH: LH Menu > Administer > User Management > Roles > Add role "Librarian" or like > edit permissions > Check the following boxes: "create Workshop content", "delete own Workshop content", "edit own Workshop content", "view revisions" > Save permissions > LH Menu > Administer > User Management > Users > Add user > fill form for Bob Bruininks including role checkbox

3. Configure the site such that you will be able to associate Workshops with other site content according to relevance to academic departments. You should be able to classify content on the site as belonging to one of the following tracks: “Astrophysics”, “Rocket Science”, “Brain Surgery”
   IDEAL PATH: LH Menu > Administer > Content management > Taxonomy > Add vocabulary > Vocabulary name (open) > Content types check "Workshop" > Save > Taxonomy > Add term > Term name "Astrophysics" > Save > Taxonomy > Add term > Term name "Rocket Science" > Save > Taxonomy > Add term > Term name "Brain Surgery" > Save

4. Create a Workshop page for the workshop “Citing Sources”, which can be described as “An overview of proper bibliographic citation formats” and which is held in the evening. (If this doesn't exactly match the Course Page fields you've created, just make something up).
   IDEAL PATH: LH Menu > Create Content > Course Page > Fill fields > Save

5. Configure your site to display a link to your Citing Sources page on the left-hand side, under the heading "Workshops"
   IDEAL PATH: LH Menu > Site building > Menus > Add menu > Menu name "classes" or like > Title "Classes" > Description up to evaluator > Save > LH Menu > Site building > Menus > Edit "Classes" menu > Add item > Path (will be supplied) > Menu link title "HIST2002" or like > Save > LH Menu > Site building > Blocks > Set "Classes" menu to "Left sidebar" > Drag "Inside this Site" to top of Left sidebar

You are the webmaster for a campus department that has chosen Drupal to manage its new website. An administrator has already performed the preliminary installation steps. You must complete the installation and begin to build your new site.

1. When Drupal is first installed, it creates an account that has full access to all system features, regardless of how the site is configured. This account has unlimited permissions on the site. The users and roles system allows you to give more limited permissions to groups of users on your site. This will be your first priority. You are now logged on as this initial administrative user. So, your first task is to
1. create an “administrator” role - IDEAL PATH: LH Menu > Administer > User Management > Roles > Add role
1. set the permissions such that this role has access to all enabled features - IDEAL PATH: LH Menu > Administer > User Management > Roles > edit permissions > Check all boxes > Save permissions
1. create a new user for yourself, make the new user an administrator - IDEAL PATH: LH Menu > Administer > User Management > Users > Add user > fill form including role checkbox
1. log out and log back in as this new user - IDEAL PATH: LH Menu > Log out link > Log in
1. For your next task we need to set permissions for the faculty, so please create a “faculty” role. Don’t set permissions for this role just yet. - IDEAL PATH: LH Menu > Administer > User Management > Roles > Add role
@COMMENT: Folding roles/permissions into next task. User #1 vs. Administrator is inside baseball, and probably not worth testing. /codyh

1. Your faculty want to have the ability to create and update "Student Resource" pages which list supplemental documents and videos available in the library. These "Resource" pages should include the title of the item, a description field that allows the faculty to use formatted text, a field to show the number of copies available, and a checkbox to indicate whether it can be checked out from the library.
* @COMMENT: Moving away from the 'Faculty Biography' content type language avoids confusion with profiles (greggles recommendation) /libsys
1. create the content entry form that provides specific web form fields for the collection of title, description, number of copies, and whether it can be checked out - IDEAL PATH: LH Menu > Content management > Content types > Add content type > Name: "Student Resource" Type: "student_resource" or like > Save content type > Content types > edit "Student Resource" content type > Add field > Field name "copies" or like > Field type "Integer/Text Field" or like > Create field > Add field > Field name "checkout" or like > Field type "Text/Single on/off checkbox" or like > Create field
2. make sure that both “authenticated” and “anonymous” users (logged on users and visitors) to your website can access (view) the content that your faculty members create @COMMENT: Since the "access content" permission is a global setting, this task is misleading. The next task should be adequate to test the permissions UI. /codyh
1. make sure that faculty members can add and edit their own Resource pages, make sure that Administrators can edit all Resource pages - IDEAL PATH:
* @comment leaving out how this is done here to see if they make the connection between users and roles based on the preceding task
1. You now realize that you need an easy way to allow faculty to classify student resource pages according to various disciplines. This can be accomplished using Drupal’s “taxonomy” system, which allows you to categorize your content using defined lists of terms. Go to the taxonomy page and make a list of research areas the faculty can select when creating and editing the Resource pages.
1. You need a list that contains the following research areas:
* “Astrophysics”
* “Rocket Science”
* “Brain Surgery”
- IDEAL PATH: LH Menu > Administer > Content management > Taxonomy > Add vocabulary > Vocabulary name (open) > Content types check "Student Resource" > Save > Taxonomy > Add term > Term name "Astrophysics" > Save > Taxonomy > Add term > Term name "Rocket Science" > Save > Taxonomy > Add term > Term name "Brain Surgery" > Save
* @comment We should probably leave out any mention of associating the vocabulary with the bio content type – this tests to see if users make the connection between content type and taxonomy
* be careful that this is set up to test drupal, and not the user. B/

1. One of the requirements of your site is to provide end-users with a menu of pages that are created on your site. /libsys
1. How would you get started with this task (test facilitator in the room) - IDEAL PATH: LH Menu > Site building > Menus
* @COMMENT: If the evaluator gets stuck, direct them to the Menu page (under Site Building) /libsys
1. Looking at the Menu page, how do you think you would accomplish this task? (test facilitator in the room) - IDEAL PATH: Add menu
* @COMMENT: Looking to see if users get confused with the "Navigational menu," which is why we do not direct them to create a new menu right off the bat. /libsys
1. Add a menu titled "Inside this Site," which can be described as "Links to pages on our site." - IDEAL PATH: LH Menu > Site building > Menus > Add menu > Menu name "inside_site" or like > Title "Inside this Site" > Description "Links to pages on our site" > Save
* @COMMENT: Note evaluator's interpretation of the 'machine readable name' field.
1. A user has already created an "About" page, located at [@TODO: PUT URL HERE]. You need to add this page to your "Inside this Site" menu. - IDEAL PATH: LH Menu > Site building > Menus > Edit "Inside this Site" menu > Add item > Path (will be supplied) > Menu link title "About" > Save
1. Notice that the menu you have created does not yet appear on the site. You must navigate to the blocks administration page and enable this menu. Make sure the menu appears on the left-hand side of the page and is situated above any other existing menus. - IDEAL PATH: LH Menu > Site building > Blocks > Set "Inside this Site" menu to "Left sidebar" > Drag "Inside this Site" to top of Left sidebar
* @COMMENT: A block configuration task or two could go here /libsys
1. @TODO: Add a task to enable users to add links to the menu (Permissions, and Menu Settings) - have maintainer test it /libsys
* @COMMENT: This task has been significantly revised (see below) so as to present a viable task, both in terms of a traditional user mental model (I need to add links to my pages on my site) and in terms of Drupal system features ("traditional" flat page/section site sections are not well supported). Rather, this task is limited to menu items only and not to the larger concept of "site sections," which may be either a debriefing question and/or a separate task. By moving this to debriefing, we learn more about how users conceive of the idea than how it plays out in Drupal. /libsys
1. One of the requirements of your site is to divide it up into various sections (“About the Department,” “Student Resources,” etc.). You can get started on setting this up by creating the “Student Resources” section and then allowing faculty to put their Resources pages into this site section. To do this, you’ll take advantage of Drupal’s “menu” and “block” system features. Menus will allow you to create a hierarchical list of links, which will give you “site sections” and blocks are containers on the sides and top of each web page that hold the menu content
* @TODO: This task may need to be rethought as configuring the menu system to work in conjunction with content forms and blocks to produce site sections is non-trivial. It will have to be reworded at a minimum. [ick] Libsys/
* @COMMENT: Agree. B/
* @COMMENT: Yes, and displaying specific blocks on specific "sections" is difficult enough to be a separate task. /elv
* @COMMENT: Organizing 'student resources' pages into a 'section' would most logically be accomplished with a view whereas 'faculty biography' (static pages rather than lists of nodes) pages would probably be better served by a menu block. The former is, I think, out of scope for these tests; views (while extremely wonderful in many ways) are probably too complex for this test. /libsys
* @COMMENT: As a result, it may be more productive to rework this task than to try to make 'student resources' page work here. /libsys
* @COMMENT: For example we might have a task that asks the Site Maintainer to create a menu block with links to key resources for faculty (ex an existing style guide on the site) /libsys
* @COMMENT: Another alternative is to use the book module to create a site hierarchy, see my comment athttp://groups.drupal.org/node/7878#comment-24287. It's certainly not the only way to create site sections, but it can be done without any contrib modules, so no worries about whether they're ported. /KarenS
* @COMMENT: Great point, Karens; the nomenclature is a bit confusing but the book/outline hierarchy is probably the most natural fit to the traditional page hierarchy model - as you point out. We'd (obviously) have to translate "book" to "site hierarchy" for the evaluators. /libsys

Debriefing Questions

• What features did you expect but not find within these tasks, if anything?
• What do you least understand about what you saw?
• What do you find the most confusing about this system?
• What does the concept "Site Sections" mean to you in the context of a departmental website?
  

Roles, scenarios, tasks for future testing

Based on consultation with the staff of the University of Minnesota Usability Lab, we narrowed the scope of this first round of usability testing to a single user role and associated scenarios and tasks. We're preserving here the work we've done thus far on those roles which are now out of scope for this round of testing, in hopes that future testing by the University of Minnesota Libraries or elsewhere might be able to take advantage of it.

Site Visitor Scenario

Open Issues

• @COMMENT: Can we fit this into the testing? Is it worthwhile? How many of the 8 evaluators should test for this target audience? B/
  • @COMMENT: Any browsing of content is really a site-specific rather than Drupal-specific issue, so this is out of scope. Registration would probably better fit under the Content Contributor scenario. /alpritt
• @TODO Expand out the following into tasks and debriefing like the other scenarios.

Summary

The goal is to create a scenario that approximates the tasks performed by a visitor to a minimally-customized drupal site. These tasks should not require any knowledge of drupal or any other web-editing or content-management platform. They should require good familiarity with internet and web browser usage.

The visitor tasks will involve searching for a web page or some information, reading a web page, seeking some specific details, commenting, registering, confirming registration, and possibly creating a profile or a basic content page.

Content Contributor Scenario

Open Issues

• Do we test the taxonomy system for these users? Which type of taxonomy do we test? freetag? both?
  • I think freetagging should be tested, also one or two other vocab-types. Perhaps one more complex vocab types, like hierarchical or multiple. B/
  • @TODO: Would you like to work up a scenario for this Bevan? It would ideally build upon the current scenarios. /libsys
• This seems very short for a one-hour session. B/
  • @TODO: We can add comment-moderation tasks B/

Summary

The goal is to create a scenario that reasonably approximates the common tasks performed by low-level users. These tasks should not require any knowledge of Drupal. Ideally, users who are familiar with any web CMS, blog or wiki platform will be comfortable with these tasks.

The contributor tasks involve adding a page, creating a link in the menu system to that page, finding an old page in the system and reverting it to a prior version and updating their username and password.

Tasks

1. You’ve been asked to create a new page with a link to this page in your site’s “about” section that describes your organization’s facilities. Create a page titled "About Our Facilities," containing the following text: “Our facilities are located in Minneapolis.”
   • @COMMENT: is "section" explicit enough. Or deliberately not? /elv
     • @COMMENT We did decide to deliberately avoid Drupal-speak in some situations where common parlance would suffice. In this case, we felt that adding a new section (i.e. a page with children) would be a common enough task in any web publishing workflow, and that by keeping the language non-Drupal, we could test the usability of some of the terms and metaphors in Drupal. Open for discussion, of course. /codyh
   • @COMMENT: I have very slightly altered the wording here by adding "creating a link" /libsys
   • @COMMENT: Just to be clear, we are not using the book/outline system here (see discussion in site maintainer) but will have a pre-configured menu block to which users are to add links. The "About" page will be created by default. /libsys
2. You realize that you made a mistake on the “About Our Facilities” page, without using your browser's "Back" button, please locate that page and correct the text to read: “Our facilities are located in both Minneapolis and St. Paul.”
3. Your supervisor is not happy with changes you made to a page you updated some time ago. She wants you to change it back to the original content . Please find this page and undo your change (set the page to a previous version).
   • @COMMENT: Does the evaluator revert the change they just made? Is this too easy? In a real-life situation you will have to choose a revision from one of several versions. In this task there are only two revisions to choose from and it's not the current one.
     • @COMMENT: The idea here was not to select a revision from the document that the evaluator just made but a page "updated some time ago" - this gets at both the usability of the revision system as well as the larger issue of locating and updating content, which more closely resembles a real world use. We could have several revisions for that page. /libsys
     • @COMMENT: Is the URL provided, or will the tester have to find (search?) the right page by himself? /elv
     • @COMMENT: Good question - the intention was to see if users could both find and revert the changes for the page? /libsys
4. @TODO: Add a comment-add task /libsys

Debriefing

1. Go back and edit the page: what does "Split Summary at Cursor" mean to you?
   1. Click this button – what do you think it does?
   2. Explain if they don’t get it and ask them what they think.
2. Scroll to the Bottom of the page. Without clicking, what do you think these links (Revision Info, Authoring Info, Publishing Options) do?
   1. Click on each (expand fieldset), what do you think they do now?
   2. What does each option in “Publishing Options” mean?

Debriefing questions could include getting user feedback on:

• Node Add/Edit features (Authoring Information, Revision Information, Publishing Options) – what do they mean?
• Expected Features that were lacking
• Overall Impressions