We are planning on doing some usability testing of the mobile prototype, below we have outlined the test plan. Since we haven't tested this prototype much, we are focusing our initial test plan primarily on finding out how they experience the navigation - not necessarily how they experience the admin theme.
Name of the project: Toolbar testing
Type of study: Moderated
Focus of the study: To get answers to the following high-level questions:
How do users use the toolbar on their mobile devices and on their desktops?
Are the users able to easily navigate to the tasks that they want to perform?
Do users understand the interaction pattern of the toolbar?
How does the experience differ when on mobile and when on desktop?
Note: Since this study focuses on mobile and desktop, to avoid order effect - half of the participants will see the mobile version and then the desktop. The order of tasks will be reversed for the other half.
Number of participants: 5 to 6 existing (Intermediate) Drupal users
Participant profile:
Existing Drupal users who have own a smart phone
Users who have a need to access/update their website on the phone
Haven't seen the new mobile toolbar design
Time frame of every session: 35 minutes
Compensation offered: $20 for external (provided by Acquia)
Pre-Session Questions
- Can you briefly describe your role with regards to your site? And how long have you been using Drupal for?
- Can you tell us about which toolbar that you predominantly use?
- How has been the experience with that toolbar? What are the things that you like the most and the least about that toolbar?
- What are your needs when it comes to accessing/updating your website via your phone?
- Besides calling/ texting, what are the common tasks that you perform on your phone?
Moderator to participant: You are about to see is a prototype. I will give you a scenario and a few tasks to perform on mobile and on desktop. But interact with the prototype as you are doing it on your site. I am particularly interested in getting your feedback about the toolbar (interaction pattern of the toolbar and anything else that you deem important).
Scenario
P1,P3,P5 - Performs this task on a mobile device
P2,P4,P6 - Performs this task on a desktop
Based on a friend’s request you decided to give a new toolbar a try for your website. This is your first time interacting with it. Please take a few moments to explore or do what you would normally do when you are trying something new for the very first time.
Question: So, what are your initial impressions?
Order of tasks for P1, P3, P5
Task 1 (on mobile)
You recently tried a new recipe and want to write that on your website. How would you do that?
Post task: On a scale of 1 to 5, where 1 being poor and 5 being excellent. How would you rate your effectiveness of doing this task?
Task 2 (on mobile)
One of your menu items says "Recipe", you would like to change that to "Recipes". Where would you go to change that?
Post task: On a scale of 1 to 5, where 1 being poor and 5 being excellent. How would you rate your effectiveness of doing this task?
Moderator to participant: Now that you have tried the mobile version, I would like to perform a few tasks on the desktop.
Task 3 (on desktop)
One of your friends, Jeff Smith wants to help with your website. You want to give him the necessary privileges to edit things that he has written. How would you proceed to do this? Note: He is a registered user on the site
Post task: On a scale of 1 to 5, where 1 being poor and 5 being excellent. How would you rate your effectiveness of doing this task?
Task 4 (on desktop)
Something went wrong and you want to take your site offline, what would you do?
Post task: On a scale of 1 to 5, where 1 being poor and 5 being excellent. How would you rate your effectiveness of doing this task?
Order of tasks for P2,4,6
Task 1 (on desktop)
You recently tried a new recipe and want to write that on your website. How would you do that?
Post task: On a scale of 1 to 5, where 1 being poor and 5 being excellent. How would you rate your effectiveness of doing this task?
Task 2 (on desktop)
One of your menu items says "Recipe", you would like to change that to "Recipes". Where would you go to change that?
Post task: On a scale of 1 to 5, where 1 being poor and 5 being excellent. How would you rate your effectiveness of doing this task?
Moderator to participant: Now that you have tried the desktop version, I would like to perform a few tasks on mobile.
Task 3 (on mobile)
One of your friends, Jeff Smith wants to help with your website. You want to give him the necessary privileges to edit things that he has written. How would you proceed to do this? Note: He is a registered user on the site
Post task: On a scale of 1 to 5, where 1 being poor and 5 being excellent. How would you rate your effectiveness of doing this task?
Task 4 (on mobile)
Something went wrong and you want to take your site offline, what would you do?
Post task: On a scale of 1 to 5, where 1 being poor and 5 being excellent. How would you rate your effectiveness of doing this task?
Post Session
- Now that you have had some time to explore this toolbar, what do you make of it?
- What did you think about the interaction pattern of the toolbar?
- Was your experience same or different on mobile vs. desktop? If it was different, how was it different?
- What was there something that was missing or confusing while using on desktop/ mobile?
- If implemented, how will this UI affect the way you work?
- What are your thoughts about having a toolbar like this, for your desktop experience?
- How does this compare to the toolbar that you are currently using?
- How would you feel if the mobile toolbar was different than the desktop in terms of design?
Participant Briefing:
Welcome, my name is .....
Outline of the session: Session will be a couple questions, a series of tasks, then a couple post-session questions.
Please think aloud
Describe your steps… what you are looking for… etc
Ask for clarification if needed, I will be be neutral throughout the test.
Please provide candid opinions about what you see. My role is of a neutral observer. So anything good or bad you say, it is not going to affect my feelings.
Keep in mind we are testing the prototype, not you As I mentioned that this is a prototype, so remember that it is not fully functional. Some of the things are working and some are not. But assume as if everything is working and go from there. Try to complete the tasks as if you were doing this for real. Spend as little or as much time as you normally would doing these task. It is ok if you cannot complete each task, and we may not get to every task
We will be recording the session, so that might share anything relevant with our team.We just record your audio and the screen, so use mouse when describing an area of screen.
Do I have your permission to record this session?
We may post the highlights from this conversation online. Your full name will not be associated with the data.
This should take about 35 minutes. The time is now ___ so we should be done around ___.
Any questions before we begin?
Feedback
As always before doing the testing we post these on the groups.drupal.org/usability for further feedback. We will revise the draft on the received feedback and start testing.
Anyone who wishes to help out testing is welcome to, although we probably do not have the funds to go beyond our 6 participant count.
Comments
We are planning on recording
We are planning on recording this as best we can, although our setups for doing this is not ideal. We hope it will be enough to give you some video recordings, on how well people are able to use this.
If you have any questions that you can't post here feel free to contact me or Dharmesh directly.
Update
Testing starts tomorrow, October 4, 2012
Session Schedule:
Participant 1: Android user on Oct 4, 2012 from 11:30 to 12:00 (EDT)
Participant 2: iPhone user Oct 4, 2012 from 1:30 to 2:00 (EDT)
Participant 3: TBD
Participant 4 -6: Bojhan to conduct the study over the weekend (Awaiting more info about participants)
If you wish to observe the session, let me know. I am also exploring ways to stream this live. Perhaps, google+ public hangout? Will keep you posted!
Dharmesh Mistry
UX Researcher | Acquia,Inc.
Testing was postponed till
Testing was postponed till tomorrow (October 5). We had some trouble with the prototype.
Dharmesh Mistry
UX Researcher | Acquia,Inc.
I would love to include
I would love to include desktop testing in this plan as well; concerns were raised by a few community members on whether or not it ultimately makes sense to have a unified toolbar experience across all screen sizes/devices or whether we should only introduce this toolbar for mobile and then have it transform into the existing desktop UX on larger screen sizes.
Some data to help support/refute those concerns would be really helpful to knowing what else is left for us to finish off the patch.
Hey guys. Awesome
Hey guys. Awesome plans!
Could we include some pre-session questions on their history and habits with mobile devices? Here are some suggestions:
I hope this would give us a strong idea of their expectations and prior knowledge.
Lewis, Very valid points! I
Lewis,
Very valid points! I always try to think in those terms as well. Unfortunately, owing to the limited session time, we couldn't incorporate all of your questions. But we did incorporate what we could in the updated plan. Thanks! Will you be at BadCamp?
Dharmesh Mistry
UX Researcher | Acquia,Inc.
Talked with @Dharmesh, and we
Talked with @Dharmesh, and we discussed the suggestions here. According to that, we now have adjusted the testplan. What we now will do is test it on both the mobile phone and desktop, where we switch during the session from mobile to desktop and vary this order to avoid any order effect.
@webchick We discussed your question. We can't really test whether people like consistency across mobile and desktop - of course they like consistency. What we can test is whether they like the mobile version and whether they like the desktop version. Which are two data points, and then the post-question of "How would you feel if the mobile toolbar was different than the desktop in terms of design?" which should give us information whether they don't care, are neutral, or really for/against it.
@lewis We tried to incorporate your question. It is in "Besides calling/ texting, what are the common tasks that you perform on your phone?". Which should capture, how experienced they are - but not necessarily provide the data points you are looking for (we should do a followup on this sometime, with a bigger questionnaire and not 35 minutes, heh). However during the testing, we will surely ask in-context questions about how they expected it to work compared to their previous experience.
Awesome, thanks! Yeah, I'm no
Awesome, thanks!
Yeah, I'm no expert at usability testing to know about which questions are leading vs. not. Just wanted to make sure since we had people lined up to do this testing we tested the full gamut of the experience. Thanks for adjusting.
@webchick @bojhan: One little
@webchick @bojhan: One little thing that I would like to add about the unified experience question, we should keep in mind - that we are asking 6 people to evaluate a design. It will uncover usability issues but doesn't necessarily represent the huge pool of users whom we are responsible to. That sort of question is more ... market research than ux research. But we have done our best to learn more about it. Hope that clarifies your concerns.
Dharmesh Mistry
UX Researcher | Acquia,Inc.
Nice one guys, thanks for the
Nice one guys, thanks for the good work and consideration.
I haven't figured out whether I'm coming to badcamp yet, I'm still talking to the organisers. Hopefully I speak to you guys properly soon, looking forward to the results.
@Bojhan If you need help with
@Bojhan If you need help with testing, please let me know. Even if it's informal testing on the side to get more participants, I'm definitely able to help.
Danielle Sheffler
Engagement Manager at Acquia
http://www.acquia.com
@Dsheffler: Thank you for
@Dsheffler: Thank you for volunteering. This round of testing is over but we have plenty of things to test in the near future. We would be glad to have you onboard! :-)
Dharmesh Mistry
UX Researcher | Acquia,Inc.
Great! Sorry I missed the
Great! Sorry I missed the date of the 5th in the comment. I won't be at BADCamp, unfortunately, but I'll be happy to help with any planning, testing, or analysis you need help with.
Danielle Sheffler
Engagement Manager at Acquia
http://www.acquia.com