Hello All,
I plan to conduct a usability study with new users to understand how they get oriented and use Drupal for the very first time. I know some of you had concerns about usability testing being favored toward the new users and not the existing ones. We have studies in mind for them as well :) It is just a matter of time and project management.
Here is a rough rough draft of the study. I would be very much interested in your feedback. Based of the feedback, I will go ahead and create a more robust screener and moderator's guide.
Drupal - Orientation & Theme Installation: Usability Study
Study Outline
- Name of the project: Drupal 7 (Orientation and Theme installation)
- Type of study: Moderated
- Focus of the study: To get answers to the following questions:
- How do first time users orient themselves when they first land on Drupal 7?
- Do understand how to perform the basic tasks like adding a slogan to their website?
- Do users understand how to install a theme and make changes to the theme?
- Number of participants: 6 participants
- Time frame of every session: 30 minutes
- Compensation offered: $25 USD Amazon gift cards (US only)
- Recruitment method: Word of Mouth, Twitter
- Study dates: January 23-27, 2012
- Other notes: -
- Researcher: Dharmesh Mistry (dcmistry)
- Total cost of the study: $150 (Sponsored by Acquia)
Participant Profile
- Has a website (not on Drupal) or has a need to create a site
- Is a site builder/ administrator/ webmaster
- Has used a content management system before. E.g. Wordpress, Tumblr, etc.
- Never used Drupal before
- U.S. only (compensation reasons)
Background Questions
- What CMS system have you used before?
- What are the things that you did for your website?
- How was your experience of you using the CMS that you use?
- When you first start exploring a new CMS, what are the things that you would want to do?
Scenarios and Task
Scenario: Let’s say that you are interested in trying out Drupal and you recently installed Drupal 7. You can login with the following credentials as the administrator.
Username: xxx
Password: xxx
Task 1: Now that you are logged-in, what would you normally do?
Observe: Do participants notice the toolbar?
Task 2: Take some time exploring how things work in Drupal
Post Task Question: So what do you think?
Observe: What is their exploratory pattern?
Task 3: Let’s say you wish to add a slogan “This is a new world!” to your site name. How would go about doing that?
Task 4: Assume that you are not happy with the current look and feel of the website and would like to change it. What would you do?
Note: Do participants explore and understand the concept of themes?
[If yes] Go ahead with the next task
[If no] Give moderated assists and understand user expectations and mental model about themes and then move to the next task
Task 5: How would you change the current theme to “Zen” theme?
Task 6: The theme you recently installed comes with a logo that you wish to remove. How would you do that?
Task 7: Besides this, what are the other things that you would normally do? (Interview based task: Allow participants to do what they would want to do)
- Consider making this task exploratory if the participants are unsure about what they will do next
Post Session Questions:
- What are the things that you liked or did not like?
- Of the areas that you saw was there anything that was missing or confusing?
- On a scale of 1 to 5, where 1 being lowest score and 5 being highest score, how would you rate this website in terms of overall experience?
- Do you have any questions for me? Is there anything else that you would like to discuss?
How was your overall experience of using Drupal for the very first time?
To the participant: Thank you for your participation.A colleague of mine will send you a $25 Amazon gift card on your email address. Could you please reconfirm your email address?
***End****
Note: I am also exploring options to stream this study to other people. Google hang out, may be?
Comments
Looking very interesting, I
Looking very interesting, I do applaud testing very specific set of tasks - rather than testing Drupal as a whole, what we tend to which requires quite some effort and often doesn't reveal specific topical issues.
In general I wonder if you want to go more beyond Task 5, depending on your timeframe - it would be worthwhile to explore more on what people do with themes. So not just switching themes, but adjust color, type of logo - really exploring the settings that are there, its not an unnatural thing to play with as beginning user, because you can quickly see the results.
Specific feedback:
Task 2) This might need some rewording, I don't think its common vocabulary to be "orienting" yourself within software. Rather focus on the activity the user is facing of "looking around" or "discovering how things work".
Task 3) Sub-title should probally be Slogan, its a more commonly used word to do what you are asking.
Task 5) The last question seems really interesting, is this something you might introduce earlier or put more emphasis on? And go more interview-based tasked with that (for those not familiar with this technique, http://www.uie.com/articles/interview_based_tasks/ ).
Great, but Needs user (role) defintion
I really appreciate these initiatives to find out how new users takes their first steps with Drupal. Its for sure something we need more of and that are very important for the work of improving Drupal.
As I mentioned on IRC, I think it is important that these test have very well defined user role to get the best possible result out of them. After all, a user is anyone that uses Drupal for their own specific needs.
In this case I believe the tasks are tailored to users who are administrators or webmasters of a CMS and that their tasks are to tweak it to their organizations needs, such as the changing slogan task and switching theme.
Does it exists a good list of defined user roles for Drupal anywhere? If not, maybe that is something that we should look into and create. Then we can always refer to it when talking about various roles.
--
/thomas
T: @tsvenson | S: tsvenson.com
Thank you for the feedback! I
Thank you for the feedback!
I am working on it on the guide and trying to incorporating the feedback. I will post it, when I am done :)
tvsenson: We do have personas for Drupal http://groups.drupal.org/node/90274 and http://groups.drupal.org/node/90849. However, I am not sure if this persona data still holds true. If I remember correctly, there was another document for the personas that I can't find. If anyone has the link handy, it would be greatly appreciated :)
Dharmesh Mistry
UX Researcher | Acquia,Inc.
3 different documents about
3 different documents about the same thing?! :)
if we do decide to update the personas, let's be sure to deprecate the non-relevant docs and point them to the correct place.
==================================
http://about.me/lisarex
Update!
Hello All,
I have updated the moderator's guide. I plan to start recruiting tomorrow, so we could get started with the study soon. Please provide your feedback! :)
The participant profile that we are looking for:
Dharmesh Mistry
UX Researcher | Acquia,Inc.
Looks good
I'm looking forward to hearing how this goes.
One thing though - the time frame of 30 mins per session seems a bit short, especially given the open ended nature of some of the tasks and interview questions. I could see sessions easily going to an hour (or more).
Cheers
Michael Keara
User Interface Systems Architect,
The User Advocate Group
Self assessment
I would add 3 questions at the beginning for a self assessment, 1-10 on website experience and have you ever done x, y, z and how long have would it generally take to do x, y z then have them try x y z in drupal.
Trying to get at how well experience in other website software translates to using drupal.
Given your small group size, you might even want to tighten up your user case definition and population sample by type of experience (ie wordpress or dreamweaver experience or a smaller range of experience )
Screener Link
Is this screener good to go?
http://arboretumd7.devcloud.acquia-sites.com/cms-usability-study
Dharmesh Mistry
UX Researcher | Acquia,Inc.