Last updated by boombatower on Mon, 2008-08-18 22:46
Project information
Project: http://drupal.org/project/uts
Status: Google Summer of Code development complete. (related release: uts 6.x-1.0-beta2)
Description
Provide a method of measuring the user experience which has been stated as a very important goal of Drupal. In Dries' keynote he detailed the results of the usability testing and stressed that we should "Continue to make Drupal easier to use." This project will provide a suite of usability testing tools that will allow data to be recorded and analyzed in order to make improvements to Drupal's usability.
Additional Reading:
Project schedule
Design and implement usability unit test creation and review interface, and provide database structure for usability unit tests. (June 14th)Create user testing environment and event capture interface. (June 28th)Design and implement pluggable API for collection of data. (July 12th)Create input plug-ins for necessary data. (July 26th)User pathLive feedbackInput recorder
Design and implement pluggable API for export, post-processing of data, and analysis. (August 9th)Provide documentation on how to use usability testing suite. (August 18th)
Status updates
2008-05-27
Basic interface for creating, editing, and deleting studies is in-place. I have stubbed out a few other features to get a feel for the interface.
I plan to have a somewhat fully featured study management interface within the week so that I can get feedback before going further into the project.
As I implement the interface I have run into several different implementation choices, especially in relation to prerequisites. I have tossed several idea around and will make a proposal in my project's issue queue.
2008-06-03
Coding on project has completely stalled as there has been allot of discussion in regards to the workings of the module. I am hoping to be able to start coding again within the next several days.
After discussion in IRC tonight we have a roadmap and I can begin coding!
I will be gone June 8 - June 15 and possible a few days longer. I should be able to make up for lost time, although combined with this last week of delay it will require some hard work.
2008-06-10
Out of town
2008-06-17
Starting to get back on track as discussions start to conclude. Most of the discussion related to the basic study/task creation interface is done so I can finish those. Hoping to get back on track after much delay with discussions.
Got a basic ideal steps recorder working then found out that wasn't what was wanted.
Started work on tasks management screen and relating them to a study. Need to confirm a few implementation level details, but should be able to wrap up interface within next few days.
2008-06-24
Study/Task management interface has been approved so I have begun work on the next phase of development ("Creating user testing environment and event capture interface").
This involves how users start/stop/interact with a study and the environment that they do it in. As I started working on it I ran into a number of unforeseen issues and start a discussion about them.
I have now completed much of the environment and session creation code as well as the interface that allows a participant to start a study. The next parts are the "Done" button and the interface to move on to the next study, pause, etc.
The project download size has doubled since last update! :)
2008-07-01
Things are moving along smoothly and I am now well into the creating the data collection API. The enabled and supported data plug-ins are enabled on the testing environment and are given the necessary functions to record data.
The data plug-ins can specify client requirements to ensure that the necessary software (ie. screen recorder, mic, etc) is available or that the browser is configured properly. If a plug-in's requirements are not met and it is flagged as a required plug-in then the user will be required to resolve the issues.
As I develop the data collection API I have been creating a live user feedback plug-in to test the API and to be the first data recorder for the UTS. This plug-in is operational in that it records user comments via an AJAX request to the server which provides a seamless interface and does not interrupt the user's testing session.
A necessary component of the proctor (module that oversees the tests) that I have yet to write is the recording of the begin and end timestamps for each task completed so that the data plug-ins can be queried with timestamps. Which leads to the last portion of the data collection API that needs to be written, the data retrieval hooks.
This module is in prime condition for an impressive alpha release, hopefully followed quickly by a beta release!
2008-07-08
I released alpha 1 of UTS and I am very pleased with how much is complete. From the release notes:
This release contains a very functional Usability Testing Suite. Users should be able to create studies and tasks, participate in a study, and analyze the results. The work-flow is a bit interrupted as some of the redirects don't function correctly, but the functionality is there.
This release contains a Live text feedback plug-in that allows participants to enter feedback using an a small dialog and send it to the server via AJAX. The plug-in needs some polishing, but it functions like everything else.
The main things left to be completed are the export/import API which will allow the analysis and data to be exported, cosmetic things like the workflow and notice e-mails, and user path data collection/analysis plugin.
It would be great if others would start writing other data plug-ins to provide feedback on the API and to help jump-start this module into a useful state.
2008-07-16
Coding has come to a stop as I wait for help deciding how to display User path comparison interface. The module is complete other than that, a few cleanup items that I will post shortly, and documentation.
I'm going to try and release a beta very soon. Talking with Bevan and Bojhan about when would be best (or what needs to be done first).
2008-07-22
Introduced base environments which allow UTS Engineers to setup an environment specifically for a study and then have it replicated for each participant. Environments can also be used by multiple studies. A have fixed a number of small annoyances and finally have redirects working which make for a much improved user interface.
I will release an alpha 3 later this week and will most likely be presenting the module at Drupalcamp Denver.
2008-07-29
Presentation went well and the recording should be posted fairly soon. I'm hoping to use part of the demonstration for an instructional video on the project page.
The suite is essential complete and I have called for a final review to see what the mentors think is left to complete or needs changing.
There is some interesting discussion on additional plug-ins that may be added after GSoC.
2008-08-05
My mentor has given a thorough review and others interested have shared their thoughts. I'm in the process of making final changes. Once complete I will update tests and write a bit of documentation.
The module still requires work to be completed after SoC, but what I have written will provide a very strong basis for that.
2008-08-12
I've been a bit busy this week, but I have completed a large portion of the items noted in the review. I hope to finish up the important items, update the tests, and write a bit of documentation before making a final SoC release.
| Attachment | Size |
|---|---|
| proposal.pdf | 174.92 KB |
Comments
Hey, Jimmy!
Really excited for this project!! :) Please let me know if there's anything I can do to help you get started.
Woot! Bevan/
Woot!
Bevan/
Bevan/
Let me put you in touch with some professionals
Jimmy, there's a bunch of people who do usability testing professionally I can put you in tough with.
They can advise you and make sure you get the resources you need.
Kieran
Drupal community adventure guide, Acquia Inc.
Drupal events, Drupal.org redesign
Drupal Log / Clickheat Map
Hey,
Just a small question how will Clickheat map help in any usability research? It only gives vague colours of where people click doesn't it? And I am somewhat worried about how you want to Drupal Log, because I would love to have usability research where the user tells exactly what he thinks but I am not sure if this happens very often, especially in online research.
Who is mentor of this project? I would love to get in touch.
Best Regards,
Bojhan Somers
Webchick and I are. PM me:
Webchick and I are. PM me: http://groups.drupal.org/user/6860/contact
Bevan/
Bevan/
Defining project scope...
I've created an issue over in the Usability Testing Suite project for Bevan, Bohjan, Jimmy, and myself to sit down and hammer out major project milestones: http://drupal.org/node/258893. Anyone else with thoughts, please feel free to chime in over there!
Morae
Bohjan told me about this program, which does usability testing. Might be worth a look at their feature demo (if you can get past all the marketing crapola) for ideas.
http://video.techsmith.com/morae/latest/demo/ondemand/index.html
Looks good
The presentation looks good. If one wouldn't need the "Manager" version (which appears a bit expensive) definitely worth a try. I know the techsmith software "SnagIt" which is definitely the market leader in Screen Capture, so the company does good stuff.
They capture a screencast of the user's desktop during a session over your LAN - you don't need a camera or anything. Hopefully painless setup. What's very important: they explicitely allow to uninstall the software from your computer, send it to someone else, who can do the same. So we would need only one or two licenses and could share the software in the community legally to whoever wants to conduct a test.
O.K. I watched the Screencast nearly all the way through. Quite impressive. One should check out the competition. But if it is the clear market leader (like Camtasia I found out being also of Techsmith), the 2000$ (I think you gotta buy the manager component and a Recorder Component for each desktop?) would be a VERY valuable invest.
I understand you record the screencast, a webcam feed of the user grimacing away while losing grip with the drupal UI ;) and also audio, which is valuable because If they want to comment, they can just speak out and don't have to write down a comment. Also I think the watching person can add a note about something he/she observed. All this data - at least they claim so - is combined and perfectly syncronized in a timeframe like in Flash. You can search for things, jump to a specific point in the recording or whatever.
Sorry for spamming like hell on your Mailbox, but the topic excites me ;) There is also a free tool http://www.taskee.com/ that does something similar, and one can get a 15-Day Trial version of morae.
Sure: the tests are still tedious, people gotta do it. But I imagine you much more easily find someone to conduct a test with this than putting up a test at a formal Usability Lab. Also it can be seen as complementary: The software could be used for focused small tests. Question: Is Views 2 UI really better than the old? Don't debate, test!
Life is a process
Life is a journey, not a destination
morae
We use Morae at the University of Baltimore User Research lab, and so far I've found it really straightforward and easy to use. And passing around the license sounds like a great idea - the more testing the better. One thing though - in my program it's emphasized quite a bit that those who build it should not test it - they're too close to it, and it's too easy to see only what you want to see. Coming up with a collaborative testing process I think is just as important as finding the right tool (probably more important).
Is that even possible?
Beccascollan as far as I know is it not allowed to pass trough licenses, it would be lovely if University of Baltimore could become more involved in the complete process of Drupal Usability testing. To provide some context and culture elements to the students, should we run this idea by Kathryn Summers and Derrick Bullock?
Although what you say, is completely true, in any context other then Drupal I would agree. However I would like to note that we don't just have 10 or 20 interfaces to test, we have over 300+ to test and getting a collaborative testing process in the scale of Drupal is simply very hard to do. If you have any ideas on how to approach this, it would be great.
As you said, the more testing the better. So even though obvious problems can certainly be missed by the developer, he is not the only one who evaluates the study results. I would love to have some kind of process in place, where the results of any usability study is simply overlooked by more people. However that is not in scope of this project, getting the culture for usability (collaborative analyses of the study) is still a big step that needs to be taken, but has to be taken without slowing down the process of usability improvements too much which as you probably know is hard. And allowing the actual developers to be part of the process of testing will make them more involved with the user and his needs, which is more important then any process.
Eigenator : I am not sure if what you are saying is correct, as far as I know only the Observer is allowed to be passed around. So the Manager still has to be a one-person kind of license. Taskee seems to be very interesting example on how to communicate usability test results.
Source of 300 interfaces?
Hello, do you have a source for citing that Drupal has 300 interaces?
Kieran
Drupal community adventure guide, Acquia Inc.
Drupal events, Drupal.org redesign
I assumed that we have at
I assumed that we have at least 300 different types of tasks that can be preformed within Drupal. From the very basic level this is assuming we have at least 300+ modules and that everyone of them will have a beta stage in which the interface will be evaluated. I am not sure how practical this however the more we push towards good user experiences testing is required, so I assumed its not wrong to say 300+. If this is not correct, please say so.
Licensing to be sorted out
@bojhan: You're right, the manager License is only for one machine. So - maybe the observer component is sufficient for a smaller test? Also, there are bundles and edu licenses as well as volume pricing... If the software is right for us, there'll be a way.
First someone should donwload the trial, which might help to find out about features of Recorder and Observer and Manager. If this helps and motivates to test, it is priceless.
Life is a process
Life is a journey, not a destination
Eigenator : I have used
Eigenator : I have used Morae in the past and it was wonderfull, however in scope of UTS what Morae does is quite different. Since essentially its a tool to help the usability evaluator, however its not a method which UTS is. Currently installed the trail on my pc again and in the given issue ques I have provided some context about Morae.
http://drupal.org/node/263972 Please take a look at this page. And look a bit into the process of the UTS.
Testing still needs to be done
Sure, the work remains, it's just a tool. But for tekky People a nice tool is always a great starting point ;)
Basically: If I sit down with some friends and ask them to test CCK node creation, my observation is totally subjective (though always horribly revealing seious flaws in the UI). If I use a tool like this - maybe a free one can do at least part of it - it's different. I can record things in Screencast, Audio and maybe Webcam. But at least Screencast. So I can watch it once more or give it to someone else - the outcome can be measured.
The motivation is different in the end: I produce (re-) usable data. It can also be more fun for the testing Person: "Oh, gosh, I take part in serious Usability science. I'm healing the world." :D Some people who would like to do testing are nowhere near a Usability Lab, and the other way round.
But still - the work is not an easy one. Alone the frustration in realizing that my beloved tool falls short to fit the needs of a "normal" user is not pleasant. But we need to go through it.
Life is a process
Life is a journey, not a destination
Hard to keep track of
Hard to keep track of discussion here and in the issue queue. If there is any issues against the workings of the module please post an issue.
Thanks to all for your interest and thoughts.
All, This page is supposed
All,
This page is supposed to be for Jimmy's (boombatower) GSoC project and the status updates. Please discuss other topics, like Morae in the Drupal community, or general discussion about usability in Drupal, to new or other existing threads. Like this one; http://groups.drupal.org/node/11992
Bevan/
Bevan/
Thanks for the status updates
@boombatower Thanks for your consistent status reports even on the slow weeks. It really makes GSoC much smoother!
--
Blog: Adding Understanding | Drupal Developer Search | Drupal Services: Brauer Ranch
--
Blog: Joshua Brauer dot com
This project sounds awesome.
I'm downloading it now and will play with it as soon as I have a gap.
Thanks boombatower for your work!!