Posted by juampynr on June 2, 2011 at 8:40am
Hello everyone!
I have just discussed within my company, www.ideup.com, about helping with further tests. We already have a usability lab with a camera and are willing to set up everything needed to bring a group of people to perform the tests.
Webchick says that there are more areas of Drupal 7 to be tested by people who does not know Drupal. Could anyone help me out to find them? Which are the top priorities at the moment?
I have been given the following links at the usability IRC channel: http://drupal.org/node/949424, http://groups.drupal.org/node/152504. We will start reading them to figure out how to start.
Cheers
Comments
That is awesome!
This is great news! Yes, there is a lot more usability testing required. Core is the priority for sure. Depending upon when and how you want to test, we can all chime in as to what could and should be tested. I need to chew more on this and understand what are the top things we should test, but I could not resist to replying to this thread - so excited! Just thinking aloud- we could test existing users (Don't take it by the word)
Thank you so much for this initiative.
Dharmesh Mistry
UX Researcher | Acquia,Inc.
Back and ready to test Drupal 7
Hi all,
First, apologies for the silence over the last days. We had to discuss within the company a few things before the go go. Here is so far what we know and what we need from you to start:
When
We want to perform the tests in 3 weeks (which means the 6th of July). During these 3 weeks we will gather a group of candidates and prepare all the resources needed.
Which is the profile needed for the test?
From what you guys told me in the irc channel #drupal-usability, the perfect candidate is someone who knows drupal but is not fond in it. We think that a good bunch of people within our company matches that, which makes it even easier.
What to test?
Drupal 7 for sure. But which areas? I have seen the Minnesota test results, which gives us an idea of what is the final output of the tests, but we need more details in terms of which areas to test and, if possible, a set of usability tests to hand to the candidates (anything off the top of your heads could be valuable).
Which resources do we need?
We have an empty room which is going to be our usability lab. We will install computers there with screen recorders and webcams. Is that OK?
What should be the format of the results?
We will evaluate the results personally, and do our best to get to conclusions that could be easily, but I think it would be nice to share all the raw output with you through a Dropbox account or similar.
I will be away until Sunday, but my colleagues at ideup! will be giving you feedback here.
Cheers!
Senior Developer at Lullabot
https://www.lullabot.com/who-we-are/juampy-nr
At DrupalCamp Twin Cities, I
At DrupalCamp Twin Cities, I presented a talk on how to conduct usability studies. Here is the link to the video which might be helpful: http://2011.tcdrupal.org/node/35
The other thing I would suggest is to decide on the goal of the study and then decide the target audience. I am not sure what is the goal of this study, if you could share that - it will be awesome! Also, more often than not I personally feel you SHOULD NOT test with internal employees. If you do HAVE HAVE to use them as participants make sure you do not disclose any of their personally identifiable information in your reports or analysis. Use P# instead. And of course, do not capture their face. (I get academic about such things! Blame it on my background :P)
For resources: I often use WebEx and it works pretty well!
For format of the results: I have documented that in the presentation.
Dharmesh Mistry
UX Researcher | Acquia,Inc.
Just watched your video, but
Just watched your video, but could not understand parts of it because of the audio quality :-(. Yet I wrote down some tips from it.
We do not have clear what to test and who to be the audience. I would like to know why some of our employees could not be good candidates, and believe that we could discuss this faster over skype, so we can speed things up (my username is juampy67).
OK about preserving their identity and not recording their faces. I will ask my colleagues about WebEx.
Finally, we would like to know if it is possible that you could send me the slides of your presentation, as I could not grab the results format correctly due to the audio issue. Maybe this could be a template that anyone else willing to carry out a usability test could use.
Cheers
Senior Developer at Lullabot
https://www.lullabot.com/who-we-are/juampy-nr
I have chatted with yoroy at
I have chatted with yoroy at IRC and we both agreed that me and my colleagues at ideup will write a list of areas to test based in what we see that the Minessota ones lack, and post it here for your approval.
Senior Developer at Lullabot
https://www.lullabot.com/who-we-are/juampy-nr
Great to hear you're
Great to hear you're interested in doing usability testing!
I posted this at http://drupal.org/node/1175694#comment-4624980 as well, but worth repeating here: If you look in the issue queue at issues tagged "Needs usability testing" that's also a good source of issues that could benefit from usability testing.
Those aren't prioritized in any way, and not all of them are necessarily well-defined enough to test at the moment, but in general those are issues that have been identified as needing some usability testing to help decide what direction the issue should go in (or whether the change proposed in the issue is worth doing at all).
They also tend to have the property that they're smaller and more focused than some of the "big" issues that were tested at Minnesota. Which means that they could also be tested more informally (i.e., maybe less important to have a full usability testing lab do those as compared to some bigger issues), but then again, having a formal lab test certainly wouldn't hurt them either :)
Sounds good...
I would be happy to share the slides. Agree that a skype call would speed up things - my handle is dmistry_usability and then we can talk at length
Dharmesh Mistry
UX Researcher | Acquia,Inc.
Hi there, My name is Marcos
Hi there,
My name is Marcos and I am the usability lab responsible at ideup!; I will be the responsible of developing and conducting the usability tests at our premises.
Juampy is now on vacation, but before leaving he gave us the objectives of the test:
We should test the default theme that comes installed with Drupal 7. Our colleague Sergio Cerrutti is even writing a book about it so I guess he should be pretty knowledgeable about it. I have gone through some of the core features and have set up a testing Drupal 7 environment so we can all play around and use it as our testing scenario with the end users.
The site currently has:
Spanish.
Automatic user account activation (so when a user registers the admin does not have to validate his account).
All core modules enabled except: openid, testing, trigger and update manager.
Basic content for each content type.
Here is a list of tests that could be given to an anonymous test user who visits the site (not an administrator, but an anonymous user).
Create an account.
Activate it.
Go to the homepage. Open an article and post a comment. Post a reply to this comment.
Open one of the pages at the top tabs. Post a comment. Edit your comment updating its subject and fields.
Add a picture to your account.
Change your password.
Post a comment in the General Discussion forum.
Vote in the poll.
Pending tests to create (add any other ones you may find):
Test the contact form: add a link to contact and ask the user to contact the administrator.
Search engine. Search for specific content.
Taxonomies
We could test the taxonomy module for administrator users.
Create a vocabulary, a set of terms and assign them to a content type.
Create a block where the terms are listed at the sidebar when you are viewing that content type.
Next steps:
We are going to do the test the environment on Wednesday 6 with 10 different users who are used to use Drupal, but they do have an intermediate-low level.
Final results are going to be available after July 15th in a pdf format with the highlights of the study.
Book nav test
Hi, this sounds fantastic - will there be any screen-capture or video of the tests?
I'm very interested in Book navigation and seeing if users understand it well (the links at the bottom of book nodes), in particular the "Up" link and if they understand the concept of "adding a child page".
Hi Jeff, There won't be any
Hi Jeff,
There won't be any problem in giving one or more (it depends on the file size) video of the sessions, the thing is that the usability tests will be held in Spanish so the videos will be in this language.
according to the topics you mention there will be no problem to introduce these tasks in our test.
thanks
That's great! We certainly
That's great! We certainly want to test the site contributors. However, I feel there is a mismatch between the participant demographics and the tasks which you want to test. I feel any web savvy user (who creates profiles and writes to forums) should be eligible for the study (irrespective of their Drupal experience) to perform the following tasks you plan to test:
- Post a comment
- Add a picture
- Change a picture
- Forum
- Vote in a poll
- Contact form
- Search
So, I would suggest if you want to focus on intermediate-low Drupal user, change the tasks or vice-versa.
Let me know what you think!
Dharmesh Mistry
UX Researcher | Acquia,Inc.
We are going to change our
We are going to change our target profile and we are going to develop the test with people who has no knowledge of Drupal at all; so the tasks for them will be more "complicated".
As a consequence of testing with this type of user, some tasks will not be done, such as administrator tasks related with taxonomy.
We are developing the final tasks and we should posted within the next hours with the final text of them.
Thanks
Based on the target we are
Based on the target we are going to test, (medium low, internet knowledge, no drupal) by now the tasks are the following:
( Please note that are traduced directly from Spanish, the originals use other words that make it a little bit more complicated to execute the task and have more text to involve the user in the context)
1- The first thing you want to do to participate in the site is creating an account. How would you sign up? once registered, you may validate your account
2- Log-in to your account
3- Once you have created your account you may customize it and you want to make these changes:
4- You have seen an article about internet television and you want to comment it.
5- You have made a comment on the article, but you are not 100% convinced of it; is there any way to change it?
6- In the article you just have commented there is a comment posted by other user; how could you reply the comment made by the user?
7- How can you participate in the poll?
8- You want to change you latest vote in the poll; is there anyway to do it
9- There is a forum in which users talk about TV; can you post a reply to one of the comments?
10- You do really find interesting to create a topic in the forum, so other users can talk about it. How could you do it?
11- You want to make an specific search, try to look for "Movistar"
12 -you do want to contact the administrator site. how could yo do it?
by now these are the tasks; how is your opinion? do you see any ohter tasks that could be given to users; please notice that are realley simple tasks but the profile we are going to use has no knowledge of Drupal.
Thanks in advance,
Tomorrow we will start with
Tomorrow we will start with the test; so if you have any comment about the tasks please let us know as soon as possible please,
Thanks
Sorry, I did not check this
Sorry, I did not check this earlier. I was away for the long weekend with no internet access. I hope I am not too late :(
My comments:
1- The first thing you want to do to participate in the site is creating an account. How would you sign up? once registered, you may validate your account
2- Log-in to your account
3- Once you have created your account you may customize it and you want to make these changes:
4- You have seen an article about internet television and you want to comment it.
- Consider giving the name of the article. E.g. You read an article "blah blah blah" and you want to share your opinion on the article. How would you do that? This way we are giving them a more specific task and avoiding the word "Comment"
5- You have made a comment on the article, but you are not 100% convinced of it; is there any way to change it?
- Are you confident or not confident that you comment is posted on the website? If you have to make sure that it was posted or not, how would you do that? The first part helps to understand about do people feel the need to check this or assume that it is worked well. Second part, addresses to your question.
6- In the article you just have commented there is a comment posted by other user; how could you reply the comment made by the user?
7- How can you participate in the poll?
- You would like to cast your vote on "Blah Blah Blah". How would you do that?
8- You want to change you latest vote in the poll; is there anyway to do it
- After some more thinking, you realize that you want change your vote on the poll. What would you do? Do you think it is possible or not possible to change your vote? Let them respond and ask. .. How would you change your vote?
9- There is a forum in which users talk about TV; can you post a reply to one of the comments?
- I would suggest: You are passionate about a particular comedy show on TV and would like to start a discussion about it. How would you do that? I am not using the word "Forum" here to understand what do the participants about the term. If the focus is not on "If users understand what is a forum". you could include the word.
10- You do really find interesting to create a topic in the forum, so other users can talk about it. How could you do it?
Isn't this the same as #9?
11- You want to make an specific search, try to look for "Movistar"
Avoid term "Search". You are looking to gather more information about "Movistar". How would you do that?
12 -you do want to contact the administrator site. how could yo do it?
Overall, this looks great! I think all these tasks are meaningful and important for a site contributor. Great job! in putting this together
Dharmesh Mistry
UX Researcher | Acquia,Inc.
Hi, Thanks for your comments!
Hi,
Thanks for your comments! We have changed a little bit the tasks in order to translate them into english; don't worry, tasks in spanish are more explined and use other words in order not to interfere with users and make them more "difficult" :). in the final report you'll have the full questions translated into english.
As for other comments; the question 10 is different from number 9; in number we want them to reply in a particular topic and in number 10 we want them to create a full topic; don't worry, in the original language is more understandable.
Tomorrow we will begin tests and we should provide you with the final report after July 15th
Thanks in advance for your support!!!
Super!
Perfecto! Good luck with the study.
Dharmesh Mistry
UX Researcher | Acquia,Inc.
Hey, How did the testing go?
Hey,
How did the testing go? Any interesting stories to share?
Dharmesh Mistry
UX Researcher | Acquia,Inc.