Content Creation UX Study Plan

You are viewing a wiki page. You are welcome to join the group and then edit it. Be bold!

Introduction

Drupal 8 wants to provide a much-improved workflow for content creators. One of the most critical screens in this is the content creation page. This issue outlines a plan for implementing a new design for this page.

The main issues that this design wants to solve are:

  • Make the publishing status of the content item much more obvious: Move the publishing state selection out of vertical tabs and position it closer to the Publish/Save actions. Provide a clear status label in the sidebar.
  • Provide clearer seperation between content and settings: Make the vertical tabs less screen-space consuming by moving them into a sidebar as an accordion.
  • Update and expand the visual language of the Seven admin theme to express all of the above in a clear and elegant form.

After some great discussions and design mockups shared in the issue queue,the community has come up with a mockup. In order to make sure we do provide a much-improved workflow, validation is required. Hence, we are about to kick off a usability study on this mockup.

This document outlines the following:

  1. Goals and methodology for this study
  2. How to find the right user for this study? (Screener to qualify participants)
  3. How to conduct this study? (A guide of tasks, and questions that we will be asking)
  4. How can you help?

1. Goals and Methodology

  • Name of the project: Content Creation UX Study
  • Focus of the project: Drupal is looking for 10 participants for 30-minute usability sessions to be conducted remotely (online) or in person. We are looking for participants who create content on any CMS-driven (Content Management System) site.

    We are looking for participants anywhere in the world.

    The study will be conducted on two parts of our audience that will run simultaneously.
    5 Participants: New to Drupal users who write content
    5 Participants: Existing Drupal users who write content
  • Type of study:Moderated
  • Length of every session: 30 minutes
  • Compensation to participants: None
  • Recruitment method:Community
  • Moderators: UX team / community
  • UX Team lead: Dharmesh Mistry (@dcmistry)

Important dates

  • Wednesday, April 11: Initial draft of test plan
  • Friday, April 13:Final draft of test plan,
  • Saturday, April 14 to May 4,2012: Conduct tests
  • Sunday, April 22:All testing ends, all results are in the analysis sheet.
  • TBD:All results have been analyzed, and final results are posted.

2. How can you find the right user for the study?

(Screener to qualify participants)

We are looking for 5 users who are content creators on a CMS but not on Drupal and 5 users for existing Drupal users. More information can be found in this screener

3. How can YOU conduct this study?

Below is a script to use.

Participant Briefing

  • Welcome, my name is X and I am helping Drupal on a project
  • This session will be a couple questions, a series of tasks, then a couple post-session questions.
  • As you go through the session, please think aloud. By that what I mean:
    • Describe your steps… what you are looking for… etc
    • Your comments are what is very important to us – we ask that you give open and candid opinions, both good and bad
    • Ask for clarification if needed, I will be be neutral throughout the test
  • Keep in mind we are testing the software, not you. Also, what we are going to see today is a prototype and it is a work in progress. But please continue to work and comment as you would expect a full version to work as.
  • My role is going to be of a neutral observer, so anything good or bad you say is not going to affect my feelings. So please feel free to give candid opinion.
  • Try to complete the tasks as if you were doing this for real. Spend as little or as much time as you normally would doing these task. It is ok if you cannot complete each task, and we may not get to every task
  • We will be recording the session, so that might  share anything relevant with the Drupal community
  • We just record your audio and the screen, so use mouse when describing an area of screen.
  • Do I have your permission to record this session?
  • We may post the highlights from this conversation online. Your full name or any personally identifiable information will not be associated with the data.
  • This should take about 30 minutes. The time is now ___ so we should be done around ___.
  • Any questions before we begin?

Study Guide


NOTE: This is a combined study guide for “New User Study” and “Existing User Study”. The questions that are specific to “Existing User Study” are marked accordingly.
 

Pre-Session Questionnaire (Estimated time: 7 mins)

  1. What is your role when it comes to content creation? Also, tell us briefly if your responsibilities related to the site?
  2. If you have to write to a blog or an article, what is the process you follow? (Note to the moderator: We are looking for high level details only, so don’t spend a lot of participant’s time on this question)
  3. Do you write the content on your blog/website editor or any other tool and paste it to your site or someway else?
  4. How many other content creators (rough estimate) do you have on your website?
  5. (If content maintainers) What is your role when it comes to maintaining content on the website?
  6. When it comes to adding/editing content, how do each of the users differ from each other?
  7. (Ask only for existing users study only) How has been your current experience with creating content on Drupal/ Drupal Gardens? What are the things that you like or do not like?
  8. (If time permits): Do you use any assistive editors while creating content? e.g. WYSIWYG, markdown, etc?
  9. Scenario 1: (Estimated time: 17 mins)


    You write for a popular food blog. Last weekend, you tried a new recipe for strawberry shortcake and you are excited to share it with your readers.
     

  10. Task 1:
    Can you describe how you would go about doing that?
     
  11. Subtask 1.1:
    You also want to make sure that your readers have the chance to able to say things regarding the recipe.
    <Note for the participant: You can create dummy content>
  12. Subtask 1.2:
    Also, You would like to add tag this content with “Dessert” and “Strawberry”. How would you do that?
  13. Probing Questions:

  14. Once you are satisfied with the content that you wrote, what would you normally do?
  15. If the participant does not comment or use, ask them about it:
    • What is “Edit Summary”? What does it do? Is it helpful or not helpful?
    • Did you notice or not notice the “Filtered HTML/ Plain Text” drop down? What do you think it does?
    • What does “Save”, “Preview” do?
    • What will happen if you hit “Save”? <Try to understand that if the expect that their content will be published by default>
    • Look at the options on the right hand side and tell us what do you think about them?
    • Take a look at each of the options at the bottom of the page and let me know what do you think about them?
    • Are they helpful or not helpful?
    • Are they clear or not clear?
  16. OBSERVE:

  • Do people understand the top part of the sidebar (what kind of information it contains)?
  • Do participants notice the sidebar options?
  • Is it clear that the sidebar contains "meta" information?
  • Do people understand that they can open and close the tabs?
  • Do people notice the submit buttons?


Post task question:

  • How would you describe your experience of this task?
  • How does this relate to the things you currently do?
  • Was there anything that was missing or confusing?
  • If you have to rate this task on a scale of 1 to 5, where 1 being poor and 5 being excellent, how would you rate your experience?
  • Scenario 2: (Estimated time: 7 mins)

    In an article that you wrote a few weeks back on chocolate chip cookies, you realized that adding one cup of walnut makes the cookies awesomer.

  • Subtask 2.1: How would you change that in the post?
  • Probing questions:

  • Let’s say that you have made the necessary changes, what would you do next? <Note to the moderator: Also, try to understand how the “Save”/”Publish” affects their content>

    Post task questions: (Estimated time: 2 mins)

  • How would you describe your experience of this task?
  • Was there anything that was missing or confusing?
  • If you have to rate this task on a scale of 1 to 5, where 1 being poor and 5 being excellent, how would you rate your experience?
  • Post Session questions:

  • How was your experience with this prototype?
  • (Ask to existing users only) Which one do you prefer: the existing one or this one? Why?
  • Would you or would you not recommend this interface as the one that you provide to your clients for their content creation needs?
  • Any other comments that you would like to share?
  • Thank the participant: Thank you again for all the comments and time! Your feedback is very well appreciated.

     

    4. How can I help?

    If you have a few hours to contribute between Saturday, April 14, 2012 and Sunday, April 22, 2012, we could really use your help!

    You could:

    • Help us recruit participants (friends or family members who are willing to spend 30 minutes of their time giving feedback)
    • Help us conduct the study: You can conduct the study yourself. It is pretty easy, trust me! Plus, it is a lot of fun. Here is the handbook on how to conduct usability studies

    If you want to help out please let us know, in this thread or over e-mail.

    Special thanks to @lisarex, @yoroy and @bojhan for the feedback. But we need more … and your feedback would be very much appreciated!  

    Comments

    @dcmistry Thanks for putting

    Bojhan's picture

    @dcmistry Thanks for putting this out!

    We are going to start testing quite soon, if anyone can help us out even by only doing one participant or referring a friend, please let us know!

    Testing

    DSheffler's picture

    I have a strong usability background and have conducted several usability tests with past clients. I'm more than happy to help!

     
    Danielle Sheffler
    Engagement Manager at Acquia
    http://www.acquia.com

    Woho!

    dcmistry's picture

    Awesome, DSheffler: Looking forward to working with you :)

    Dharmesh Mistry
    UX Researcher | Acquia,Inc.

    We can help

    cherringland's picture

    I can definitely recruit a few people to participate from our team/company, including at least one person who uses Drupal daily, and one who use a ahem different CMS.

    Updates

    dcmistry's picture

    Hello @Cherringland,

    Any updates? You can send me the e-mails (with time zones, and country) of the participants to me (dharmesh.mistry@acquia.com).

    Dharmesh Mistry
    UX Researcher | Acquia,Inc.

    We can help

    cherringland's picture

    I can definitely recruit a few people to participate from our team/company, including at least one person who uses Drupal daily, and one who use a ahem different CMS.

    Nice!

    dcmistry's picture

    @Cherringland,

    Fantastico! You have no idea this is going to help us so much with the project.
    So, the next steps would be:

    • Qualify participants for studies using this screener
    • Once you have qualified participants, enter their non personally identifiable information in this spreadsheet
    • If possible, also ask for the tentative times that they are available for the session. It makes scheduling slightly easy :)
    • I or someone from the team will contact the participants for the study

    Dharmesh Mistry
    UX Researcher | Acquia,Inc.

    userfeel.com - willing to offer 5 tests

    yannisc's picture

    Hello Guys!

    I'm the owner of netstudio, a Drupal development services company in Athens, Greece. We have developed www.userfeel.com, a remote usability testing platform.

    While remote usability testing is not the same as normal usability testing, it can provide quick and easy to get feedback. The main difference is that the test is not attended, that means you have to provide the whole script in advance and have the user follow the steps by himself. Also, there is a 15 minutes time limit on the tasks that the user should complete. The final result is a video with the user's actions and voice feedback and also written answers to 4 questions.

    I'm willing to offer 5 sessions for free if you think that would be helpful for the task described. I'll need the scenario somewhat adapted to remote usability needs. I can do the screening based on your criteria and try to find the right testers among our testers panel.

    Please tell me if I should proceed with this.

    First remote usability test ready to view.

    yannisc's picture

    I got ahead and screened the users. Found 7 that match our criteria and posted the test for them.

    Here is the first result: http://www.userfeel.com/result/NGY4Y09ESXdOdz09NGY4Yzdm

    I'll have the rest ready in the next hours.

    Yannisc, You rock! I am

    dcmistry's picture

    Yannisc,

    You rock!

    I am coordinating with other contributors for this study. Can you tell us if they are existing or new users?

    Dharmesh Mistry
    UX Researcher | Acquia,Inc.

    2 of them are existing users,

    yannisc's picture

    2 of them are existing users, the others are not.

    The one I posted already is not an existing user.

    Super!

    dcmistry's picture

    Sweet!
    Can you update this sheet when you get a chance?

    Dharmesh Mistry
    UX Researcher | Acquia,Inc.

    Dharmesh, I updated the sheet

    yannisc's picture

    Dharmesh, I updated the sheet with the first 4 tests.

    I've also uploaded them to YouTube for easier (maybe) commenting and sharing:

    http://youtu.be/PNJCHYGZHFQ
    http://youtu.be/MW5EeSTLTYM
    http://youtu.be/g7Ciz0mxeMM
    http://youtu.be/UYtVuc5a82M

    (I've added the URL's in the sheet as well).

    @ÿannisc That was fast,

    Bojhan's picture

    @ÿannisc That was fast, wasn't aware you where already testing (we have actual test instances, which where clean). Next time please catch us, before proceeding because I saw you also didn't change the test plan to fit remote testing (observer notes, where mixed with questions).

    Either way, great to see you have already shared the results! I am watching them now, the actual videos are sometimes hard to see, is there any way you could improve the quality?

    Bojhan, I have adapted the

    yannisc's picture

    Bojhan, I have adapted the scenario to remote testing as I saw fit, but ofcourse I'm open to improvements to this one to use with the next tests.

    Please also point me to the clean installation to use that also.

    As for the video quality, unfortunately, we cannot have it better. Compression is applied for faster uploading on the tester's side.

    thanks,
    Yannis

    @yannisc Its mostly 13, that

    Bojhan's picture

    @yannisc Its mostly 13, that might need some changes - I have noticed most of them answered this after completing the task and often skipped over parts. You could consider leaving out that one.

    See the sheets for the clear installations to use.

    Contributors moderating

    dcmistry's picture

    Contributors moderating sessions,
    I have created an analysis sheet to put your notes from the sessions. Right now it is a private document but will ultimately be made available.
    Please drop me a line if you want to access the notes.

    Dharmesh Mistry
    UX Researcher | Acquia,Inc.

    New users for testing

    thorandre's picture

    I have some contacts that are content editors that are new to Drupal. Two of them can do this testing with me, if still needed?
    We've have done several usability tests with out past clients.

    Let me know! Seems like you've got what you need in the Participant Sheet, but I am happy to give some extra input on this if needed :)

    Thor Andre Gretland | Front Kommunikasjon | Twitter: @thorandreg

    Thorande, Great! We are

    dcmistry's picture

    Thorande,

    Great! We are looking for participants who haven't used Drupal but use other CMS. We have enough of participants who use Drupal. Would you know of any such participants?

    Dharmesh Mistry
    UX Researcher | Acquia,Inc.

    Sounds great! Unfortunately I

    marvinrmurphy's picture

    Sounds great! Unfortunately I have missed such nice opportunity so anyone can say me that when it will be held in future?

    Usability

    Group organizers

    Group categories

    UX topics

    Group notifications

    This group offers an RSS feed. Or subscribe to these personalized, sitewide feeds:

    Hot content this week