Attached are the results from the usability study I did for at the University of Baltimore. This was for the Research Methods class with Professor Kathryn Summers - we did usability testing with an eyetracker and looked at the results.
What I found most pressing was that all of our participants had a lot of frustration due to not understanding that the administrative functions overlayed the website itself. It made seemingly simple tasks much more difficult for them to grasp. I had one case where a participant explained to the moderator after the test was over that some CMS's have an admin that overlays the site, yet still didn't get that that was what he had been looking at for the past hour. I think any push towards integrating more contextual clues or help and creating more visual feedback that a user is within the administrative portion of the site (like a dashboard) will be a great help to new users.
I'd be happy to post the session videos - any recommendations on where to put them would be appreciated.
| Attachment | Size |
|---|---|
| DrupalUsabilityResearchReport.pdf | 810.57 KB |
Comments
Post videos on service of your choice
Then link to them here, and I'll cross post here.
http://drupal.org/handbook/customization/videocasts
Cheers,
Kieran
Drupal community adventure guide, Acquia Inc.
Drupal events, Drupal.org redesign
No attachment?
No attachment?
This report is very
This report is very interesting and insightful. I hope we will see some improvement on Drupal usability based on this work.
Great work!
Hello, Becca Scollan
Time and time again I find the work comming from the University of Baltimore of great value. I will take the time to get all of these issues into the que and hope that Catch can help with supplying patches on the major issues that can easily be fixed, so that we can take more time before D7 to fix the major issues. We have already noticed that the lack of an actual admin interface is a major issue, however we would need a culture change within Drupal for a shift towards an admin interface to happen.
It would be great if you could post the videos on a video service that allows big resolutions, such as google video. If this video solution doesnt supply enough detail, please contact me and I will upload the full files to my web server.
It might be intresting to note that I see you used the Handbook of Usability Testing by Rubin Jeffery, that Dana and Jeffrey just brought out a new version with lots of new information on recruiting, techniques ect.
Best Regards,
Bojhan Somers
This is really, really good.
This is really, really good. I recognise a lot of the same issues coming up from Minnesota (admin vs. site, page/story, losing content, clicking 'help' links and getting disoriented, teaser splitter, menus) - so that's both good and bad. The report itself is very, very easy to read. Hopefully we'll get some patches in soon.
Thanks everyone! I have an
Thanks everyone!
I have an opportunity to do more work as an independent study in the fall, so if there are some longer term projects that need work, please post. I'm kind of interested in the whole idea of mental models right now (Indi Young's new book) - looking at how people perceive building and maintaining a website and possibly doing some paper prototyping with wireframes based on the results. I'm open to other suggestions as well.
I will be at the UPA conference this week and will get to the videos next week.
Just drop by
Hey, Becca Scollan
I just finished reading the book, I got a ton out of it. Alan Coopers book just doesn't cover it in-depth at all, and Indi Young did it great, lots of more conversational type of writing. It might be interesting to apply more application type usability techniques to get a better understanding between the different user groups (Jared Spool has a podcast on this, and a few reports that cover it slightly) and anything you gain of insight of mental knowledge with that would be great.
I am not the one to say, which part of Drupal needs usability testing, but I am sure that if you drop by a few weeks before fall we can create a better scope on what we need testing on.
For us the challenge is breaking the usability problems into action-able lists of stuff to do, as some of the major problems are completely overhauls of how Drupal tends to do stuff. So culture is definitely an issue on our part, but I am sure in time we will work most of that out.
I know that on the UPA confrence , Pawan Vora will be doing a seminar on web applications so it might be interesting.
Best Regards,
Bojhan
Thanks for this
Thanks for sharing -- confirms what we have found in the field training for a long time -- people are often very confused by the admin interface sharing the same template as the front end.
I will post this into our Scribd open source CMS group if you have no objections: http://www.scribd.com/groups/view/8399-open-source-cms
rico!
water&stone
a full service digital agency
www.waterandstone.com
Reviewing usability studies
Finally got round to reading the whole report. I must say this is excellent, and very clear. I particularly like the 'error level severity' as it helps point out where the biggest issues are and where we should prioritise.
In issues where only one participant is confused, is this enough information to take action on? For example, in issue 9 a participant expected to see 'home page' used instead of 'front page'. If we changed this to 'home page' how do we know other participants would not be confused by that new term? In this case I can guess that 'home page' would work better, but I don't know for certain. Even if we repeat the tests, would we know that we have solved the issue when the number of participants is so low?
My closest reference to this kind of report is from psychology papers which rely heavily on checking for statistical significance. That's perhaps not appropriate for usability studies, but I don't want to get in the habit of making changes when the evidence isn't fully qualified; that would be wasted effort and potentially confusing to current users.
Perhaps there are some resources I should read to help me get up to speed on reviewing usability reports? I feel a little ignorant at the moment.
-
www.alanpritt.com
Map usability results to Drupal issues
There have now been two independent studies both with 8 users. I think we've got good data.
The immediate task now is map problems found in both studies to Drupal issues, I think catch is helping to lead this effort. I am going to carve out time this coming week.
Kieran
Drupal community adventure guide, Acquia Inc.
Drupal events, Drupal.org redesign
I'm not sure front page/home
I'm not sure front page/home page is an issue in itself. But I don't think 'promoted to front page' is a particularly good description. Maybe 'show on front page'? 'Sticky at top of lists' got some raised eyebrows at UMN if I remember correctly.
Alpritt statistical statistical significanse
Alpritt I would like to state that statistical significance is something I usually just ignore, a lot of people state it but its hardly an desired goal in a usability study. Obvious problems are found by fairly any audience and experts don't really have trouble understanding beginners terminology or are even bothered by it.
To see if our improvements are good, we have to run a new usability study on the made improvements. The number of participant of usual usability studies is 8 to 12. And almost all usability books vouch for this amount, some even suggest less users to improve and test again to find new issues. Almost always you see patterns in user behaviour due to usability issues within these numbers, in some cases 1 user is even more valuable then the other 7 due to how the participant can communicate without obviously communicating.
If you really want to find out more, I suggest to listen to Jared and Christine, who also state its just a buzz word.
http://www.uie.com/brainsparks/2007/10/22/usability-tools-podcast-statis...
I don't think there are any resources on how to read usability reports, since usually in organisation, people don't acctually read them. People prefer actionable issues and to discuss and see the issues with the usability team, ie presentations. Thats why I suggested Becca Scollan to post the videos, since that will help communicating the issues way more then a report.
Most usability firms look for statistical significance when they are finding out the main tasks of the target audience. Then numbers of at least 30 participants are thrown around, however I am unsure if you could define that as really statistical significant. A trend (good move!) in the usability community is to make qualitative studies as quantitative as possible.
Great Work
It is a pleasure and a pain to read this. Like someone pointing you to the mess on your desk that you been long intending to clean up...
Well, the Admin section/Actual page issue can be a bit cut down by using a separate Admin theme, which I ALWAYS do. But still content creation remains frontend theme, so this part, that took up most of the tasks as I understand, will remain a hurdle.
I cannot really remember clearly my first impressions of the interface. At first install I liked it, because it was not as cluttered as Joomla's or Typo3's. But this was way back in 4.7 time. Don't know how I would have reacted to the Admin dashboard page. When it came, I was past my initial struggles and found it a great improvement. But how long till users find the "hide descriptions" link. Personally took me more than half a year =8-0
Life is a process
Life is a journey, not a destination