Drupal 7 Usability Study at Google

We encourage users to post events happening in the community to the community events group on https://www.drupal.org.
You are viewing a wiki page. You are welcome to join the group and then edit it. Be bold!

What’s the study?

The purpose of this study is to gain insight into how non-engineering, tech-savvy users interact with the Drupal content management system when they first begin using it. We have two groups of users: the first group (February 1) will start with vanilla Drupal 7 and we will be studying their conceptual understanding of Drupal themes. The second group will start with the Drupal 7 as well as a number of modules relevant to the tasks pre-installed. For the second group (February 3), we are more interested in UX issues related to specific core/contrib module interfaces. This project aims to raise awareness of Drupal usability issues to a broader sector of the community. The sessions will be livestreamed and available for anyone to watch by using Livestream.com and a Google+ public hangout.

Who are you?

My name is Becky Gessler and I work for Google. Previously I ran a Drupal business and specialized in usability testing and interface design. I love Drupal... and I remember how hard it was to learn way back then and I have watched countless clients struggle through it. D7 has made leaps and bounds in terms of UX, but it is still a struggle to start learning Drupal. I would like to help lower the learning curve that myself and so many others have faced.

Study Outline

  • Number of participants: 8
  • Session length: 1 hour
  • Compensation offered: Google Open Source clothing
  • Recruitment method: Word of Mouth, Internal marketing
  • Timeline: February 1 and February 3 2012
  • Google is sponsoring this study

Participant Profile

  • Google employees comfortable with technology in their personal and professional lives
  • Don’t possess any significant engineering experience
  • Very minimal to no experience with a CMS or Dreamweaver
  • Genuinely want to create a basic website (so we can see them use Drupal in a realistic way)

See this document for detailed user profiles based on the background questions we asked them.

Script for February 1

In this study, we are testing the participant's understanding of Drupal concepts like:

  • Creating content
  • Adding a content type
  • Adding fields
  • Modules (adding, enabling, configuring)

February 1 users
These users have been chosen for their preference of information-heavy websites that stress organization, categorization and blogging capabilities.

These are the tasks jenlampton, bojhan and I have come up with for our Feb 1 user group. The "XYZ" is there because each user has a slightly different content type they will be working with, based off their answer to the background question about the kind of website they would like to build. In this way, the tasks will be slightly customized for each user in order to simulate a realistic task they would face when building their website.

  1. Orient yourself. to get them comfortable at first with exploring
  2. Put up some background information for your site. to see if they can add a page
  3. Create an article about your subject, tag this content with relevant keywords. introduction to taxonomy and tagging
  4. By default, Drupal has two flavors of content: page and story. Create a new one of these for XYZ [insert content type]. testing to make a content type
  5. You’d like to add the tagging functionality you saw on Articles to your new XYZ. add an existing field to the new content type
  6. In addition to tagging, you would like to categorize the content of [XYZ by difficulty - don’t use tagging, use a controlled set of options] test taxonomy to create a controlled list vocabulary and add the field to content type
  7. So right now XYZ has a title and description. Let’s say you also want to be able to associate a date / time information make relevant to their XYZ]] with XYZ, like XYZ. How would you do this? introduce modules, can they find it, can they install it, enable, etc.
  8. You want people to be able to send you an email directly from a form on your website. Can you see if Drupal has a way of doing this? see if they can find a module in the list and enable and configure it
  9. You don’t like that all the URLs on your site end in node/1234. See if you can find a way to have nicer URLs for all your content. install and configure pathauto

Script for February 3

In this study, we are testing the participant's understanding of Drupal concepts like:

  • Creating content
  • WYSIWYG module
  • Adding a content type
  • Adding views, using simple Views UI and complex Views UI
  • Configuring image styles

February 3 users
These users have been chosen for their preference of media-heavy websites.

These tasks are drastically different than the Feb 1 test. Keep in mind all modules are pre-installed and enabled, and we are not trying to test conceptual Drupal understanding here; instead we are going to get users into the interfaces and see what they do. My moderating will be much more leading and direct to get them to the right places. (However once they get there, they will be left alone).

  1. Take some time to orient yourself and verbalize your impressions. to get comfortable at first
  2. Create an article about your vacation using the sample text provided and tag this content with any relevant keywords. hopefully an easy win to get them going
  3. Style the text in your article using the options available to you. see how they interact with WYSIWYG
  4. Use the rest of the sample text in new articles, and tag them with different keywords. they need to make multiple articles for the View they will build
  5. Okay so let’s say you want to create a list of your articles and have them sorted by the date they were created, from earliest on. Let’s navigate to the Drupal Views interface in order to do this. Try to get this list to appear in the tabs on the top, next to home. to test the Views simpleUI.
  6. By default, Drupal has two flavors of content: page and article. Create a new one of these for photos. This is what you will use to add your vacation photos to your website. You will also need to add the ability to add a photo to your content type. I will very much guide them through this process, since we have a lot of data on this from Feb 1
  7. Create some photo posts for your website using the images provided in the folder on your desktop. setting them up for the second View
  8. One image is too big. How can you make sure images are always re-sized to fit the page? to test image styles... after they find it, I will say, okay great you changed it, but how about if you wanted to make it a custom size?
  9. You want to create a photo gallery for your vacation pictures. Let’s go back to the Views interface to accomplish this. testing Views complexUI. they will save the view with simpleUI and by default it will only show teasers, so they'll have to go back in to complexUI and play around
  10. You don’t like that all the URLs on your site end in node/1234. See if you can find a way to automate the creation of human-readable URLs for all your content. to test pathauto again. this module will be installed but not enabled.
  11. You want people to be able to send you an email directly from a form on your website. Can you see if Drupal has a way of doing this? to test contact form
  12. Now let’s say that you would like to add the same tagging functionality that you used when you created an article to your photo content type. How would you do this? back to fields
  13. Great. Now you want to add a view that lists your same articles, sorted by date they were created, but only showing those tagged with ‘beach.’ Also try and get it to appear as a tab up here next to “home” another View if we have time

Recorded Videos!

You can watch the videos of all 8 participants on YouTube from this playlist:
http://www.youtube.com/playlist?list=PL1B7D6E805CE3A5A0

Users 1-4 went on Feb 1
Users 5-8 went on Feb 3

AttachmentSize
drupalgoogleUX-sm.gif6.18 KB

Comments

Super Awesome!

dcmistry's picture

This is super duper awesome, technicka! I am very glad to see usability effort progressing.

Overall, the script looks good. My two cents:

  1. "Please describe the website you would like to create. Explain its purpose and what users will be able to accomplish/learn on your site. Include any functionality you wish to include (photo gallery, video archive, ability to tag posts, etc.)"

I am wondering if we should link this to build a website that is similar to the one they already have? My thought process behind this is that this will allow the participants to easily associate and have a robust task requirement structure as opposed to them thinking on the fly.

  1. "What is most important to you when choosing a platform to create a website?"

I find these questions very useful. Perhaps, consider renaming it to “If you are creating a website, what is your criteria in choosing a platform?” (Keep in open ended and if needed, then share the option list). Also, consider adding a follow-up question “How does this criteria relate to the website you intent to create?” (Trying to understand if their criteria is different if they are building a blog site vs. a revenue generating site)

  1. It might be good to consider some post sessions questions as well.

  2. Also, in my studies (for tracking purposes), I have started tracking tasks (and studies) in terms of effectiveness, efficiency and experience on a 5-point Lickert scale with 1 being the lowest and 5 being the highest. In order to save time, the effectiveness and efficiency ratings are rated by the researcher/moderator. Although not very scientific, they are a good baseline. But I follow a rule based on the definitions of what means to be “effective” and “efficient”. For “experience”, I ask them.

  3. Once you have finalized the tasks/scenarios, would it be possible for you to post the wordings as well?

I would be happy to help in every little way that I can. Let me know!

Dharmesh Mistry
UX Researcher | Acquia,Inc.

Awesomeness in the making

eigentor's picture

This sounds well thought-out.
The pool of potential participants inside your company sure is an advantage. In our companies, mostly everybody is drupal-trained...

I guess dcmistry is the one most up-to-date what other UX studies are done or planned. So he can best keep track of what areas of Drupal have already been tested and which could use more testing. So if Dharmesh has no objections to the general scope and participant profile, this should be fine...

Starting from the bottom, people that are new to Drupal should be the best study participants as long as we have not paved the road past the first steps perfectly. Taxonomy and Blocks are probably to be considered "intermediate" site builder stuff you do when you have successfully created your first nodes. I think earlier studies tackled more basic tasks (or participants never got past the basic ones), so this sounds good to me.

Life is a journey, not a destination

What we don't know

tkoleary's picture

This is great and will definitely pull out some data that we need to know in order to improve usability and increase adoption of Drupal. The one thing that I think is missing is an opportunity for the user to clue us in to what is missing. In going through the test, at times users are going to think "I wish this worked like 'x' product" or "I wish this had feature 'y'", or "It would be easier if I could do operation 'z'". If we only ask questions related to functionality that already exists we miss the opportunity to capture important new ideas.

Backgroud question feedback

jenlampton's picture

I wonder about the question "If you have any familiarity with HTML, CSS, and/or JavaScript, please describe your familiarity here." Instead I would ask about familiarity with builiding websites, and let them specify if they know HTML/CSS/JS or if they have Used Dreamweaver or other website building tools.

This should also help weed out participants

mcG73's picture

If he or she starts using development lingo and talking about language specifics they may not be the best subjects for gathering newbie information.

Updated Background Questions

technicka's picture

Hi everyone,
Thanks so much for your feedback so far. I have updated the background questions section of the main wiki. My top priority is to have these questions sent out ASAP. I currently have 10+ interested possible participants, so it looks like we’ll have the luxury of choice depending on what types of profiles we’d like to account for.

@dcmistry (Dharmesh): I would like it if participants actually had a website in mind they would like to build, be it similar to an existing one or not. Based on the results of the survey, we can see what is most appropriate. I agree that post-study questions would be helpful.

Please give me your feedback about these questions so I can send them out. Thanks!

Hi Technicka. I've been

McRick's picture

Hi Technicka.

I've been thinking about collecting various snippets from usability tests into a "Worst of Drupal" compilation. Any chance we would be able to use the video from these tests to support this effort? The idea is by regularly getting these videos out in a easily-digestible format, we might get developers thinking more about usability in their work on Drupal. The first step is to collect existing video into a few videos. Then, in the near future, organize some usability testing efforts (including our own). I don't want to highjack your post to promote this effort, but I did want to get you thinking about this as a possibility for your videos.

Thanks!

UX Director, Bluespark

Videos will be recorded

technicka's picture

Hi McRick,
The videos from the studies will be recorded and made available, so there is the opportunity to make a reel of clips. I'm also hoping people will watch the studies live.

Great opportunity and the

yoroy's picture

Great opportunity and the plan looks fine already, thank you for taking this on.

How do you see analysis and reporting findings back to the community?

I too hope other people will watch live, ideally this would be people who could help with analysis, prioritizing issues etc. There's a lot of work in getting the sessions translated into the right actionable issues in the queue. Lets find smart ways to share the workload.

Steep Curve

m.shahriarinia's picture

I am in! Drupal has a steep curve to jump start low level coding for folks. We gotta make it easier. I think starting with a multi-page dynamic form is a good starting case study for most of the businesses.

Study Dates and Times

technicka's picture

Hi everyone,
Just wanted to give an update on study times. As of right now, participants are scheduled for Wednesday, February 1 from 12pm-5pm PST and Friday, February 3 from 12pm-4pm.

More details to come on how to watch live, but it will likely be through a public Google+ hangout.

HOW TO WATCH LIVE (Wed, Feb 1)

technicka's picture

I have posted details about how to watch the study in the main wiki section (at the bottom). There will be 4 sessions tomorrow from 12-5pm PST.

I have also updated the script (tasks we will be running through and goals) and I created a document with detailed profiles for each participant so you can have some more context.

Spread the news! Remember the last 3 sessions will be on Friday, February 3 from 12-4pm PST.

Time Zone conversion

eigentor's picture

Europe gets their time zone converters out...
So this would be 20:00 UTC/GMT for the start time?
Which would be 21:00 for MEZ, even if that is not an international Standard time notation.

Life is a journey, not a destination

Time Conversion

technicka's picture

12:00 PST is 20:00 GMT

Yay!

dcmistry's picture

Looking forward to this! I would be happy to be involved in the analysis and prioritizing issues. I am traveling a bit this much so I will be a little time constrained. But nevertheless, always happy to help!

Dharmesh Mistry
UX Researcher | Acquia,Inc.

Post Survey Questions

technicka's picture

If anyone has any suggestions for questions we should ask participants after they've taken the study (sent out in another form) please let me know. Remember we will also be in #drupal-usability during the sessions to discuss.

Will try to watch

User Advocate's picture

I'm very interested to see the live stream but may have to watch the recordings after the fact (have meetings already scheduled).

Michael Keara
User Interface Systems Architect,
The User Advocate Group

Great work

mcfilms's picture

Thank you technicka for opening this up to the community. Certainly it would have been just as easy for Google to do this as an in-house project. Hopefully we'll all benefit from this. Originally I thought I'd watch for a few minutes. I ended up sitting in on three sessions.

Here are a few Post Survey Questions I thought of:

Which task did you find most challenging and why?
Do you think creating a website using Drupal will be harder or easier than you anticipated before the session?
Can you think of a specific instance where a function appeared "broken" or did not work the way you anticipated?
Was there any particular function names or jargon used in the Drupal interface that confused you?

By the way, will these survey results be published as well?

@mcfilms: thanks for your

technicka's picture

@mcfilms: thanks for your suggestions about questions. We will try and work these into the sessions tomorrow through a survey.

The survey results and videos will be published. We'll also figure out a time to have a discussion forum for anyone that wants to participate who watched the studies.

WATCH LIVE FRIDAY (updates)

technicka's picture

Hi everybody,
It was awesome to see how many people watched the studies on Wednesday (oftentimes Livestream had 100+ users!!). Let's continue this trend for tomorrow's studies!

I have updated the user profiles in this document:
https://docs.google.com/document/d/1i7dJRnvj8Rk9zDwYCnTpQ7FNJ4TswFcJpyzW...

I have added the task script for tomorrow's sessions to this wiki page. Keep in mind this script is drastically different from the first and our goals are different (read more on wiki page).

We will be streaming live in the same fashion from 12pm-4pm PST, likely with an additional participant, so really till 5pm PST. We'll be back in #drupal-usability.

Thanks for watching!

recordings of sessions

nadiakirilova's picture

Hi technicka,

I am very interested to watch the recordings of the studies but I wasn't able to watch them live - are the recordings going to be available soon? I was looking through this page but can't seem to find a link.

It's a great effort with this study that you are conducting - it's going to be interesting to find out what users find difficult to use in terms of interface and see how it can be improved. I will use Drupal in our project for the development of a platform that facilitates open design, so I am a novice user of Drupal but I have used other CMS tools. Drupal is different from all the ones that I have used in the past and I am learning to use it myself at the moment. So I am going to be following up on how things develop.

Thanks in advance.

Nadia

Recorded Videos have been Posted

technicka's picture

Hi everyone,
I have posted all 8 videos to a playlist on YouTube:
http://www.youtube.com/playlist?list=PL1B7D6E805CE3A5A0

I look forward to the analysis that lies before us! I will be working with Garen and Jen this week to create a higher level summary that we can then all collaborate on and turn into actionable items.

@technicka thanks for all the

Ranko's picture

@technicka thanks for all the effort and getting the sessions on YouTube. This is very interesting to see.

Detailed findings now available!

technicka's picture

Hey everyone,
Check out this handbook page for a table of the detailed findings from the study:
http://drupal.org/node/1427940

We need help putting these problems into issue queues!

Usability Report posted!

technicka's picture

To view the high-level usability report from this study, see:
http://groups.drupal.org/node/218959

Also, if you are here at DrupalCon Denver, be sure to come to our core conversation session on Wednesday at 1pm to learn more about how we ran this study and a breakdown of the results.
http://denver2012.drupal.org/content/user-experience-open-source-how-gal...

Usability

Group organizers

Group categories

UX topics

Group notifications

This group offers an RSS feed. Or subscribe to these personalized, sitewide feeds:

Hot content this week