Help Needed with Formal Usability Test Development

Note: the ongoing "official" testing plan document can be found here: http://groups.drupal.org/node/7929

As announced here http://drupal.org/node/204667, Drupal will undergo formal lab usability testing this coming February. This project builds on the already impressive work of the usability group and elsewhere. We feel honored to play a part in this ongoing effort.

The testing process begins with the identification of high-level goals; target audiences; common user tasks for these audiences; and a set of debriefing questions for the evaluators that we recruit for testing (users who test the product are called “evaluators”). It’s critical to the success of this project that we scope the goals, target audiences and tasks appropriately.

The U Libraries testing team has drafted a goal statement, target audience list, task list and proposed a variety of debriefing questions as a preliminary measure towards moving this process forward. We now need the Drupal community to help set priorities; propose and refine tasks; suggest debriefing questions; and to help out with various other tasks.

Please take a look at the following rough draft and offer (specific) feedback towards the identification of goals, target audiences, tasks and debriefing questions.

Thanks!

As Drupal has yet to undergo any formal usability testing, our primary objective is to identify audiences and craft scenarios that will enable us to get a baseline understanding of Drupal’s usability for common tasks. As a result overall testing priority will be given to Content Contributors and Site Maintainers but some tasks may be reserved for the Admin role where administrative tasks would not require a great deal of prior training or experience (ex installation process ;) ).

Scenario Overview and Usability Goals

Scenario Overview
This round of testing will focus on the needs of users who maintain relatively "standard" corporate (or departmental) websites. This is the general context in which we identify users and their needs. The audiences that follow represent users who would play a role in this process.

[Placeholder Usability Goals]
@COMMENT: I just pulled these out of you-know-where to get the ball rolling. /libsys

• New users with minimal web design/development experience should be able to perform basic content management tasks with moderate direction. Such tasks include adding pages to the site, adding links to pages within site menus, placing pages within the site hierarchy and commenting on content.
• New users with moderate to advanced web design/development experience should be able to perform critical system maintenance operations. Such task might include administering users (adding them, adjusting access privileges, etc), creating data entry forms for content contributors (see definition below) to manage dynamic content, building hierarchical sites, adjusting the look and feel of sites, locating and managing versioned content, etc.

Audiences:

• Content Contributor
  • We envision the content contributor role to be that of an authenticated user with permission to create content such as pages, menu items, and comments.

• Site Maintainer (editor/subadmin)
  • We envision the site maintainer as an authenticated user with access to manage user permissions and content, including Blocks, Taxonomy, Menus, Comments, and all Nodes.

• Admin
  • We envision the Admin as an authenticated user with access to all site settings, responsible for installation and configuration.

Content Contributor Scenarios

Summary

Our goal here is to create a scenario that reasonably approximates the common tasks performed by low-level users. These tasks should not require an intimate knowledge of Drupal. Ideally, users who are familiar with any web CMS, blog or wiki platform will be comfortable with these tasks.

The contributor tasks involve adding a page, creating a link in the menu system to that page, finding an old page in the system and reverting it to a prior version and updating their username and password.

Task Scenarios

1. You’ve been asked to create a new page in your site’s “about” section that describes your organization’s facilities. Create a page containing the following text: “Our facilities are located in Minneapolis.”
2. Now you need to create another site section.
3. You realize that you made a mistake on the “About Our Facilities” page, please go back and correct the text to read: “Our facilities are located in both Minneapolis and St. Paul.”
4. Your supervisor is not happy with changes you made to a page you updated some time ago. She wants you to change it back to the original. Please find this page and undo your change (set the page to a previous version).

Contributor Debriefing

1. Go back and edit the page: what does”Split Summary at Cursor” mean to you?
   1. Click this button – what do you think it does?
   2. Explain if they don’t get it and ask them what they think.

1. Scroll to the Bottom of the page. Without clicking, what do you think these links (Revision INfo, Authoring Info, Publishing Options) do?
   1. Click on each (expand fieldset), what do you think they do now?
   2. What does each option in “Publishing Options” mean?

Debriefing questions could include getting user feedback on:

• Node Add/Edit features (Authoring Information, Revision Information, Publishing Options) – what do they mean?
• Expected Features that were lacking
• Overall Impressions

Open Questions:

• Do we test the taxonomy system for these users? Which type of taxonomy do we test? (ex freetag? both?)

Site Maintainer (“subadmin”, editor, etc)

Summary

The goal of these tests is to create a scenario that reasonably approximates the common tasks performed by a site maintainer. These tasks will require some familiarity with web content management systems in general, if not Drupal specifically. We will endeavor to capture testers’ initial impressions of various aspects of Drupal, and then follow up with such instruction as is necessary to complete the tasks. Tasks will include enabling modules, setting up new users, adding content blocks, establishing menus and vocabulary configuration. Possible secondary tasks may include theme and site information configuration.

Site Maintainer Task Scenarios

[SCENARIO OVERVIEW] You are the webmaster for a campus department that has chosen Drupal to manage its new website. Your systems administrator downloaded Drupal and performed the preliminary installation steps. You must complete the installation and begin to build your new site.

1. When Drupal is first installed, it creates an account that has full access to all system features, regardless of how the site is configured. This account is similar to the “root” user account in an operating system. For a variety of reasons, it is considered a “best practice” to avoid operating and configuring Drupal while logged on as this user. Rather, it is better to create Drupal “roles” (administrative groups of users) and to then set specific permissions for those roles. This will be your first priority.
   1. You are now logged on as this initial administrative user. So, your first task is to (a) create an “administrator” role (b) set the permissions such that this role has access to all enabled features© create a new user and (d) put this user into the administrator role.
   2. Log out and log back in as this new user.
      1. We’ll need another role for your next task, so go back to the user administration area and add one more role, the “faculty” role. Don’t set permissions for this role just yet.

1. Your faculty want to have the ability to create and update biographies of themselves on your website. Biography pages should always contain the faculty member’s name, a picture of the faculty member and some biographical content.
2. How would you get started?
   1. First, you’ll need to create the content entry form that provides specific web form fields for the collection of name, biography and picture data.
   2. Next, you’ll need to make sure that both “authenticated” and “anonymous” users (logged on users and visitors) to your website can access (view) the content that your faculty members create.
   3. You’ll also need to make sure that faculty members can add and edit their own biography pages. [leaving out how this is done here to see if they make the connection between users and roles based on the preceding task]

1. One of the requirements of your site is to divide it up into various sections (“About the Department,” “Student Resources,” etc.). You can get started on setting this up by creating an “About” section and then allowing faculty to put their biography pages into this site section. To do this, you’ll take advantage of Drupal’s “menu” and “block” system features. Menus will allow you to create a hierarchical list of links, which will give you “site sections” and blocks are containers that hold the menu content [ick]
2. [NEED INPUT HERE] This task may need to be rethought as configuring the menu system to work in conjunction with content forms and blocks to produce site sections is non-trivial. It will have to be reworded at a minimum.

1. You now realize that you need an easy way to allow faculty to choose areas of research specialization on their biography pages. This can be accomplished using Drupal’s “taxonomy” system, which allows you to categorize your content using defined lists of terms. Go to the taxonomy page and make a list of research areas from which faculty can select their own.
   1. You need a list that contains the following areas: “Astrophysics,” “Rocket Science,” and “Brain Surgery.”
      1. [We should probably leave out any mention of associating the vocabulary with the bio content type – this tests to see if users make the connection between content type and taxonomy]

Debriefing Questions

• What features did you expect but not find within these tasks, if anything?
• What do you least understand about what you saw? – What do you find the most confusing about this system?

Open Questions:

1. We have yet to test these scenarios and do not have a sense for time to complete them. Testers (“evaluators” in usability parlance) will have about 1.5 hours each.

Comments and Questions

• Test Site Configuration Suggestions will also be Solicited
  • Depending on the target audience, we are suggesting varying levels of pre-configuration w/different installations. This will make the tests more meaningful.
  • An installation profile might be helpful promote scenario reuse and benchmarking
• We are particularly interested in the idea of trial run test feedback, in having Drupalers run through the scenarios with their own users and providing us with feedback
  • We will have a D6 demo site up and running sometime in the next week or so

A few notes about the testing program:

• We will have 8 users for the entire testing program an industry standard number, more or less.
• Each evaluator will have 1.5 hours to complete the aloted tasks.
• We currently have split the testing plan 50/50 between the “Content Contributor” and “Site Maintainer” target audiences.

Key dates:

• January 14th – Kickoff Meeting
  • This meeting will include a first pass review of tasks, audiences, etc.
• February 11th – Scenario/Prototype Review
  • In depth look at scenarios and prototype walk through
• Feb 26th-27th – Testing
  • Eight Evaluators will test the product
• Feb 28th – Issues Analysis
  • A review of testing outcomes