UB: Test plan

We encourage users to post events happening in the community to the community events group on https://www.drupal.org.
Bojhan's picture

As you might know, we will be spending a week at the University of Baltimore doing a usability test (http://www.bojhan.nl/usability-testing-at-university-of-baltimore/). We intend to focus this study more on solving the problems that have previously been identified by testing at the UMN and UB. While we have fixed quite a list of issues, there are still allot of critical usability issues we would like to see fixed in Drupal 7 we hope everyone will help out in order to get these patches in Drupal 7 before we start testing persons - so that we can see whether the solutions we came up with work with people.

The goal is to test Drupal with 8 persons, of whom 5 beginning users and 3 intermediate users. We will ask the beginning users to preform a series of task that are similar to the other studies,

The current layout for the test plan :

Screener
These are the questions that will be asked, before they enter the study - we will be looking for 5 persons who have no previous experience with Drupal, while the other 3 would have some experience working with Drupal (not to extensive) .

  • Where do you work?
  • What is your job title?
  • How long have you worked in web development?
  • What CMS' have you used in the past and do you use currently?
  • What type of websites do you build or maintain?

Scenario
We will build a scenario based on the specific person we are testing, if the interview we do with him reveals he is in the process of switching CMS for a music site. We would work the scenario towards, building a website - but if this is not the case we have a scenario.

Your asked to build a website for a conference, where information will be posted about the conference like schedule, speaker bios and local bars.

Interview goals

  • What kind of sites are they currently building?
  • General information about their work process and resources?

Tasks
We chose tasks that will allow us to research patches and problems we have encountered before, main issues are; finding functionality, finding content and terminology.

Task 1 (Open task) :
Password (register their admin account)

Task 2
Put up some general information about the conference.

After they publish the content, ask them what they think they're looking at is, if this isn't clear then let them know they are looking at the website itself.

Task 3
Now you want to create a navigation link on the the left hand side of the page to the general info about the conference.

What would you call this? Where would you expect to look?
[allow participant to attempt task]
[guide them to site building > menus]
what is this page? Is this where you can complete the task? Where/how do you expect to do it?
[also need a follow up question]

Task 4
Looking at the site right now, is there anything you want to change?
Change the visual look of your site.

Task 5 [roles]
The conference organizers want the speakers, after they registered and logged in to be able to post information, but have no other access to the site.

What would you call this? Where would you expect to look?
[guide them towards usermanagement > roles]
[after ones has created a role]
How do you expect to allow them to post information?
[skip if to hard]

Task 6
Create a schedule page.
Create a navigation link on the right hand side of the site.

We currently have only six tasks, because our experience learns that people have a lot of difficulty using Drupal at first. We might make changes to the tasks, depending on the specific participant or on something we learned from the participants.

Feedback
We are looking for feedback on the test plan, are there any issues that have been patched that we should look into, or anything that might need to be patched before we head into the usability testing. All feedback, really.

Also, we would like you to get in contact with us if you live in Baltimore (or near) - to meetup and maybe help coordinating.

Comments

task 5

greggles's picture

We did task 5 in Minnesota and the feedback was basically that roles/permissions were fairly simple for people to understand. It was the one part that most participants handled without problem. Unless there is some specific new UI for that area that you want to test, I suggest that task is removed and we find some new area to test.

--
Growing Venture Solutions | Drupal Dashboard | Learn more about Drupal - buy a Drupal Book

Task 1

catch's picture

I'd like to test the new installer, but we can't make them download Drupal unzip it etc.

But... what if we do all the basics for the Drupal installation, and start them on the "enter your database details" screen - with a cheatsheet for what they need. This would then get them to the 'set up your site' page - and we can test the password checker there. Since this is the first step for everyone who uses Drupal, would be nice to see how we do - a little bit easier than reality because they don't have to handle file permissions etc., but those are operating system/hosting specific anyway.

Greggles: Is there anything

Bojhan's picture

Greggles: Is there anything you'd like us to test? Or that you experienced that needed further testing? Since, roles/permissions is something quite essential, we'd need to be at the point where we arnt suprised anymore by the testing.

catch: We could, I was wondering though if we should incorperate that with the intermediate tests.

my two farthings worth

Shannon Lucas's picture

Important: Confirm with Becca whether or not her university requires her to file all questionnaires with her institutional review board (IRB). When I was in graduate school, everything involving human participants had to go through the IRB if I wanted to publish any findings. Bojhan, if you're planning to derive an academic project or paper from this, check with your university as well.

What is your job title?

Titles are not descriptive since there is no consistent definition across the industry. Consider asking them to briefly describe their job role.

How long have you worked in web development?

Not everyone who manages content management systems is a developer. Some may write code but not consider themselves developers. If the question above is changed to ask about their job role, this question could become How long have you worked in this or similar roles?

What CMS' have you used in the past and do you use currently?

Don't use the acronym. Spell out "content management system" instead. Perhaps something like What content management systems (e.g., Drupal, Joomla, WordPress, Stellent, Alfresco) do you have experience with?

There are no details on the setup of the experiment. Is this a base Drupal install or are certain other modules installed? Is there a database already configured? What are the steps to reset the environment between tests? What are your approximate criteria for screening applicants after the questionnaire has been submitted?

Task 4
Looking at the site right now, is there anything you want to change?
Change the visual look of your site.

It's not clear what the scope of this task is. Are you trying to determine just if they can figure out how to choose a different theme from among those that are already uploaded, or do you want them to go through the more realistic process of uploading a theme and then selecting it? If the former, perhaps the task could start with you have just uploaded the theme 'Acquia Marina' and wish to use it for the conference site...

How does Task 6 differ from what the participant is being asked to do in Tasks 2 and 3?

I think this is a good start, but there needs to be more detail in the tasks. There should also be some explanation of what you expect to learn in each task. Think of this as any other research project. Each step of the experiment needs to have a reason for being there. What variation do you anticipate between your inexperienced and experienced users? Will the experiment include a different set of tasks (such as the installer) for each group?

Good work!

beginning user...

joe.murray's picture

I realize I'm way too late on this, since this round of testing is about to be reported at DCDC... but here goes anyway, since I'm hoping there will be further work on usability.

I'm just wondering about the beginner / intermediate user distinction.

From my perspective as a consultant, I'm interested in seeing Drupal usability improvements not so much for those configuring and customizing sites, but for those who are using sites built with Drupal. I like to distinguish between what an authenticated user - a site visitor, if you will - can do, and what a content administrator / moderator is required to do. Site or system administration tasks, which change the structure of the site, are not as important as candidates for usability testing, at least for beginner / intermediate users.

For Drupal to be great, it isn't the first few things done with a core only install that are important, but figuring out how to use a system with a small number of pre-configured cck's, some of which are user generated content that need to be moderated, and some of which have comments enabled that also need moderating. Pulliing in some embedded media from YouTube and Flickr, aggregating some content ... these are things we need to be doing usability testing on. I realize this involves some key contrib modules and not just core, but these are central elements to a large number of Drupal sites.

Joe Murray

Joe Murray

editor role

gaele's picture

This is not a beginner you're talking about (admin can be beginners too) but a specific role: the editor or content manager role.

That said I completely agree: a big +1 to testing usability for editors.

UB results

Usability

Group organizers

Group categories

UX topics

Group notifications

This group offers an RSS feed. Or subscribe to these personalized, sitewide feeds: