Edit: This proposal has been approved by community vote as of 06 November 2008. See http://drupal.org/node/331079
Previous usability testing at the University of Minnesota(UMN) and the University of Baltimore(UB) highlighted many issues with Drupal's usability, resulted in a number of solutions, and did a great deal to raise the profile of usability testing within the Drupal community as a whole. While follow-ups to these sessions have been mentioned, as yet, no further labs have come forward to donate facilities. This KDI proposal is to guarantee formal testing of Drupal for the next two years, so that the profile and momentum already achieved within both the Drupal development community, and the wider user experience community as a whole can be maintained and built upon.
We would like to apply for funding for two, one week long, testing sessions. Three days of formal testing, then a three day sprint afterwards to collate the data, prototype solutions and share findings with the whole community.
These would be conducted at a usability lab or equivalent facility, and would include key people in the Drupal community (including developers, usability experts and designers) being on site for the duration of testing. The sessions would be held six months apart, during the week before Drupalcon. This would allow for sufficient time between testing to develop solutions for some of the issues found, and is the most likely time to get 10-12 Drupallers in the same place at one time.
This proposal lies at the heart of the goals of the Knight Drupal Initiative: “To enable more people to enter the digital conversation by lowering the technical barriers to entry“. Usability improvements in Drupal's install system, content creation and management, and other core functionality will lower the barrier to entry dramatically for people without previous experience in web publishing and development.
The focus of this proposal is to identify and analyse common usability issues in the Drupal core application. Solutions to these would then be built as part of Drupal's normal development process. Usability testing provides solid, evidence based methods to push for improvement in Drupal interface patterns and workflows. By enabling key members of the Drupal community to attend formal testing sessions, we will also increase the level of expertise in this area through first hand experience.
The two previous lab tests already resulted in smaller scale ongoing testing using similar techniques, and a higher level of awareness of these issues and methods within the community at large. The usability group on drupal.org at http://groups.drupal.org/usability has over 450 members and has become very active. The newly formed 'User Experience team' held a four day sprint at DrupalCon Szeged and has been holding regular online meetings since then. Several usability improvements have gone into Drupal 7 early in the development cycle, with these changes usually reserved for after 'code freeze' - still some months off for this release.
This funding proposal is for a one year cycle of testing: two sessions, six months apart. This provides us with two imporant points of reference to measure our current status and progress made. These lab tests will inspire ongoing small scale testing and lead to future testing after the initial project is completed (via documentation, building relationships with university departments etc.).
Actual development of solutions to the found issues is likely to take from as little as two weeks to several months - but funding of this development work, other than the three day post-testing sprint, is explicitly outside the scope of this proposal.
Since the Drupal project does not have a fixed release schedule, and we would need specific individuals to be available for the week of testing, it's difficult to provide specific dates, however:
February 2009
August 2009
Each session would be held in the week before Drupalcon, this benefits scheduling, availability, and ensures maximum impact trough a comprehensive followup at Drupalcon following each lab test.
There would also be a requirement for writing up tasks prior to each testing session, which would be conducted remotely in association with the facility. Including these pre and post production phases, this project will take approx. one month for each test.
Although we would conduct the entire process openly with the Drupal community, the team for implementing the project would be as following:
Nathaniel (catch) Catchpole
Regular contributor to Drupal core since 2007, participated in usability testing at the University of Minnesota and the resulting presentation at Drupalcon, has authored and reviewed many of the core patches resulting from it.
Bojhan (Bojhan) Somers
Usability Consultant since 2006, Google Summer of Code mentor for the Usability Testing Suite and involved in Drupal usability initiatives since early 2007.
Roy (yoroy) Scholten
Interaction designer, contributing interface design, art and copy since early 2007. Initiated the User Experience Sprint at Drupalcon Szeged.
Brad (beeradb) Bowman
Contributor to the Drupal project since 2007 in the areas of support, contributed module development, core patches, and community organization. In addition to his software development duties, he helps run in-house usability testing for Aten Design Group.
At least one member of this team would attend each session. Additionally, we aim to ensure that the branch maintainer for Drupal is in attendance (currently Angela (webchick) Byron), Dries Buytaert (Dries), and other members of the core development team and 'the UX team' up to a maximum of 10-12 attendees per week.
Deliverables would include:
- A test plan document including a description of the purpose and problem statements, participants, methodology and the test script.
- A list of all usability issues found, ranked by severity.
- Detailed descriptions of the main issues posted on groups.drupal.org/usability and in the Drupal issue queue. The issue queue is an effective method for instigating usability improvements in Drupal. Over 50 issues were posted to Drupal.org directly connected to UMN usability testing. (see http://drupal.org/project/issues?projects=3060&text=umn%20usability&stat...)
- A presentation highlighting high priority recommendations with footage of participants for the European and North American Drupalcons and drupal.org frontpage.
- Summary reports of design solutions found at usability sprint gatherings at Drupalcon posted on groups.drupal.org/usability. Usability sprints at Drupalcon will bring together the Drupal community to tackle individual issues in more depth and are based on the pattern started in Szeged by yoroy.
- An online archive of video feeds, eye tracking data and all related reports and presentations.
- A simple guide to effective low-fidelity usability testing that summarizes the process of planning and carrying out a test to promote testing throughout the Drupal community.
Besides these concrete deliverables, these lab tests will very likely inspire more usability related projects through, for example, Google's summer of Code.
(more on Potentially spin-off funding proposals to develop new interfaces based on the research - KDI, Summer of Code projects etc. here?)
All costs in US Dollars:
Per test:
Hire formal usability lab, equipment + staff for 3 days: 10,000
Cost of meeting venue: 1,500
Flights and board for 12 attendees: 15,000
Partial reimbursement of 4 attendees, 1,000 a week: 4,000
Administration, technical set-up, 80 hours at 50: 4,000
Funding for testing participants ($100 x 8) 800
______
Per sprint: 35,300
*2 tests
Total for this request: 70,600Combined with Drupalcon, this would be a full two weeks away from work for all attendees. The earnings reimbursement is a buffer to ensure people who need to attend are able to do so without company sponsorship - for example self-employed, students etc. We feel it's better to account for this up front then remove the component later. And while the flights and board are the largest individual expense, developer education, both directly, and via feedback to the community, is one of the primary benefits of these lab tests - witnessing people using our software first hand is a unique experience that can profoundly affect the direction of the project. We would expect the individuals attending each session to rotate somewhat over the two sessions, so that 12 or more higly involved individuals in the Drupal project can attend.
Comments
demonstrated pay-off?
This is great. Something that struck me as missing, however, was an indication of how we plan to demonstrate that the team and the process has been effective.
If improvements have been made from a previous set of tests, testing the improved features in a later set of tests would provide some validation both to our benefactors and to the UX team. I'm not sure that enough would get implemented on the 6 month mark to accommodate testing the improvements / fixes, but there should be enough by the 12 month mark.
I think it would give KDI some warm fuzzies if the proposal mentions how we plan to hold ourselves accountable and what metrics we'll use to say "hey, we improved Drupal."
improvements
I think there's two aspects of improvements we'd hope to encourage with this.
As you say, within either 6 months or a year, we'd hope that the development community would have some solutions to x number of issues arising from usability testing. Certainly that's been happening with the issues from UMN.
For the first round of testing in this run, I'd hope we'd go back and re-test areas which we think will be improved in Drupal 7 - early next year that's likely to be the new help system, the welcome page, finding content after you've created it etc. - hopefully some other things as well. That will help us to verify the changes we've made in core already. So in each iteration - test things which have been recently 'fixed', and test things which haven't been tested yet.
Having said that, I think the 'success' there is primarily the feedback from the testing rounds (did things improve? are there new problems?), since this proposal isn't going to fund implementation beyond some basic scoping. Which leads me onto the second main deliverable:
By tying usability testing rounds to Drupalcons, and ensuring that we have a presentation at each Drupalcon based on those findings (as we did in Boston), we can almost immediately communicate usability issues (and successes) to hundreds of Drupal developers and designers in person. AThis gives us a 'State of usability in Drupal' presentation twice a year. While it's not directly affected by the proposal, the UX sprint at Szeged led to more than 80-person hours (15-20 x 4 + code sprint) spent on UX improvements in core during the conference week - having usability testing and time to filter issues before the UX sprint would help to shape the issues which are covered, and should lead to solutions quite quickly - in Szeged we had patches for one of the issues quite quickly, and others are moving on quite rapidly since then - all four issues came out of usability testing at UMN and UB.
In addition retesting solved
In addition retesting solved usability bugs in successive tests, as Shannon suggested, we can also track issues and patches resulting from the testing and sprints in the Drupal Issue queue. It may be worthwhile pulling statuses of these issues into an external system or spreadsheet so that an overview, statistics and metrics can be generated and turned into reports.
A report after the sprint could say; 100 issues were created resulting from this testing and sprint; 10 have been committed, 20 others are RTBC, another 30 have patches, 3 or more people are discussing solutions in another 20 (Three or more people usually generate a healthy discussion leading to a viable solution).
A report one month out could say; Of the 100 issues, 30 have been committed, 10 more are RTBC, another 50 have patches all remaining 10 tickets have healthy discussion.
A report 3 months out could say; Of the 100 patches, 60 have been committed, 20 more are RTBC, 15 others have patches, 5 others have been closed without commits, postponed or marked duplicate.
Etcetera.
Another issue that was raised in IRC was that the UX team might isolate themselves resulting in patches not getting the attention they deserve. There are two important points that make sure this will not happen;
Both Drupal 7 maintainers and core committers strongly support Usability testing and Usability improvements. Dries has been saying this for a long time. Both Angie webchick Byron and Dries attended the UMN testing. Angie pulled together a significant part of the slideshow and community rallying that resulted in the community getting behind the efforts of the UMN testing and patches getting crafted and committed.
As part of this proposal current Drupal core HEAD committers will be able to attend the testing, which results in their appreciation of the Usability bugs discovered and their raised awareness and attention to usability issues and patches.
Bevan/
Bevan/
Excellent fit for KDI
This is a tremendous proposal and a very valuable proposition. In addition to the relatively modest sums requested per-test, the tie-in to Drupalcons and corresponding focus/energy will multiply results tremendously.
5 Stars.
Drew Gorton
Gorton Studios
I'm really excited to read
I'm really excited to read this proposal. I mean, REALLY excited!
I had hoped to have an impact in the State of Drupal Usability through my Season of Usability project, but I quickly learned that no one person can ever have a significant impact on the State of Drupal Usability. I learned that it must come from the Drupal Community as a mass -- through the collective effort of a group of individuals; the community itself.
If this proposal gets approval, Usability will become as much a part of Drupal core development as code style, patch review, documentation and testing. The proposal will bring the Drupal Developer and Drupal Usability communities together a way that will have a lasting effect and impact on Drupal core development.
To date, the strength and speed of the Developer community has outweighed the Usability community so much that usability improvements have barely been sufficient to play "catch-up" on old Usability bugs and bad UIs. This proposal enables a change in the Drupal community that will allow Drupal start making usability improvements and becoming easier to use in leaps and bounds, rather than small minor improvements.
Further, many very valuable members of the Drupal community are unable to make as much time available for contributions to the Drupal project and Drupal Usability community as they would like to and the Drupal project can benefit from; This seems to be especially true for members of the Drupal Usability community. Funding for usability related projects would enable these talented individuals to contribute their sorely needed time and skills.
Regards,
Bevan/
Bevan/
Great proposal and badly
Great proposal and badly needed. It may have taken until D7 to start taking usability seriously, but a formal process like this can have quick payoffs in addition to the obvious long-term effect. I think tying this to drupalcons is a very good idea, because some of the success of this proposal hinges on developers awareness. One shouldn't underestimate how quickly developers with little or no UX background learn to avoid basic pitfalls once they've seen people struggle with their designs.
Code testing?
Although this proposal is stated to be for usability testing, are there plans to extend it to code testing and SimpleTest development in Drupal core? Either way, my vote is in!
not simpletest
While I personally think a KDI proposal for some of the things testing.drupal.org is doing might work well, it's pretty far out of scope for this particular proposal.
Ongoing Usability/UX Program is a Natural for KDI
Great work, UX team. The Knight initiative seemed to me like it was written with an ongoing usability program in mind when I first heard it proposed - wonderful to see a thoughtful plan put forward to that end.
It's so easy to get caught up in meta-discussion about the overall UX strategy, which is important but can also be a little paralyzing. A sustained effort of testing and reporting out is bound to surface much fruitful discussion about and positive change in the area of user experience and to generally keep the ball rolling.
5 Stars from me.
This would be awesome -- and
This would be awesome -- and much needed.
I would extend the proposal a bit so we make sure there are actual deliverables in terms of recommendation documents, presentations, and maybe even some issues with proposed mockups.
deliverables section
Here's how I would re-write the deliverable section:
A test plan document including a description of the purpose and problem statements, participants, methodology and the test script.
Detailed descriptions of all issues found during testing posted on groups.drupal.org/usability and in the Drupal issue queue. The issue queue is an effective method for instigating usability improvements in Drupal. Over 50 issues were posted to Drupal.org directly connected to UMN usability testing. (see http://drupal.org/project/issues?projects=3060&text=umn%20usability&stat...)
A presentation highlighting high priority recommendations with footage of participants for the European and North American Drupalcons and drupal.org frontpage.
Summary reports of design solutions found at usability sprint gatherings at Drupalcon posted on groups.drupal.org/usability. Usability sprints at Drupalcon will bring together the Drupal community to tackle individual issues in more depth and are based on the pattern started in Szeged by yoroy.
An online archive of video feeds, eye tracking data and all related reports and presentations.
A quick and dirty guide to usability testing summarizing the process of planning and carrying out a test to promote testing throughout the Drupal community.
I don't think spin-off proposals are officially a deliverable, but they should be included somewhere in this proposal. Make sure you describe what the different programs are in more detail.
I used to work for a grant-giving organization and watched many proposals go before a panel. I think it's important to remember that a) the people making the decisions are really intelligent but b) they have no idea what you're talking about.
Thanks all for reviewing so
Thanks all for reviewing so far. Good to see everybody is positive about the plan. We will be adress your main points in an updated proposal. Being more conrete on actual deliverables and metrics for actual progress are sound suggestions. Thanks again!
Sweet
This is a great idea. Glad to see this usability movement is gaining momentum. I'd love to help out.
Great idea
This is a great idea and would be a huge step forward.
--
Blog: Adding Understanding | Drupal Developer Search
--
Blog: Joshua Brauer dot com
need bigger budget for planning, admin, and follow-up
I think there needs to be a significantly increased number of hours in the budget for planning and administration, and for monitoring of the outcomes by the organizers. The proposal is great otherwise.
cut to one year
We've just cut the scope of the proposal down to one year (i.e. a session prior to each Drupalcon in 2009) - since the overall budget is now half as big, we could probably look at increasing the administration component a bit - and a lot of administrative work is likely to be concentrated around the first sprint before we get the hang of it, so having just two sprints means there's less lee-way for this in the overall budget.
We need this
For context, I'm speaking from the perspective of someone who would be taking the results from these tests and helping to design – and in some cases implement – solutions to fix the major issues.
From the previous tests we have plenty of data to be going on in order to start fixing issues, but my big concern is always making sure we actually fix them rather than just change things. So I would like to make sure there is an emphasis on retesting what we have previously failed at. I know this has already been mentioned above, but it is worth repeating since this is going to be very helpful for determining if we are doing a good job. I also think that this will help motivate designers and developers to make future improvements by providing a clear deadline to get solutions to a state where they can be tested and by providing feedback on success.
I don't believe the testing period should place too much emphasis on designing solutions unless it particularly makes sense to make refinements while testing is still occurring. If time and money is to be spent, it is going to be best spent on getting great data to work from in the months between testing sessions. Categorisation on the severity of the issues and commentary on why the user may have problems with certain UIs would be more useful. Great reports and eye tracking videos are what we need from this more than anything. The better the reporting, the more the wider Drupal community (i.e. those who cannot attend themselves) can make use of it. If there are privacy concerns for the participants, it would be good to be aware of them ahead of time so that we can focus on making the data as accessible to the community as possible, while working within that constraint.
If you have room to expand the scope, it would be great to spend resources on investigating the requirements of our market more. We have a good sense of the benefit of this from the drupal.org redesign and it would be good to get some wider (perhaps less formal) feedback from our wider user base. We don't really know the scope of people who use it (and especially those who try to use it and give up).
Coinciding with the Drupal conferences is a superb idea. I'm a little unclear as to whether you will try to place them geographically together (within reason), or whether this will be possible. But either way considering we don't know in advance when the next version of Drupal will be released, coinciding with Drupalcons makes the most sense for establishing when these tests will happen.
Most of all I will that this isn't just a great idea, it is absolutely essential that we continue our formal usability testing in the coming years. Without it the quality of Drupal will suffer greatly as we try to fathom where our usability problems are without knowing for certain whether our fixes have helped or were the correct problems to solve in the first place. As Drupal matures and adds new and powerful features, it is only becoming more important to make sure the people who could benefit from it most can actually use it without extensive training. Without this testing there is no doubt we will miss problems that scare potential users away. With it we can make it an indispensable tool for a much larger population.
-
www.alanpritt.com
There are many ways to test
This is actually one of the major advantages of conducting tests at drupalcons where many of the implementers are present. A usability test doesn't only have to be conducted once and revisited 6 months later, there are other forms of testing as well. One that goes particularly well with this scenario is a RITE test, where rapid iterative designs are mocked up with the aid of the project developers, and each iteration can be tested again with users. If you're really scrambling you can test two iterations per day (though one is more realistic), and by the end of drupalcon you have a design that has been revised several times with data collected after each revision. It's a very powerful way to make a lot of progress in little time.
RITE
I used the RITE method when I did the usability testing for Thief 3, and it worked out really well. I conducted tests a couple of days apart which allowed the developers to get changes in between tests. Usually, though, changes from one test session didn't make it in until after the next test session, so fro tests 1, 2, and 3, changes from test 1 usually made it in for test 3.
Another advantage to this approach was that it gave me the opportunity to get the kinks out of the test scenarios and the test systems by the first two tests.
More Info:
Using the RITE method to improve products; a definition and a case study [.doc]
Rapid Iterative Testing and Evaluation RITE [.ppt]
Summary of the Thief 3 testing (butchered by the editors).
User-centered design in games (ACM Citation only - you'll need access to the actual journal archive to read this one).
This proposal complements less formal usability
Rapid iterative testing has been suggested -- along with the suggestion in the thread below that only 5 test subjects are needed. Certainly any developers implementing usability improvements should organize less formal usability tests on their own.
With very little cost and effort, developers can use less-formal techniques to quickly conduct iterative testing. With just a few hours of effort, you can gather invaluable feedback by having a few test subjects try out a new version of a form or feature that you are re-designing. Jakob Nielsen's "discount usability engineering" along with RITE and other methods all apply here.
We should all be doing less-formal usability testing as we work on Drupal features.
There is still a huge need for the formal usability testing proposed in this Knight initiative. This is a great proposal and I hope it moves ahead to the Knight Foundation to receive funding. Formal sessions with a larger number of users are needed for the whole community to objectively evaluate Drupal at key points in the release cycle when we combine all of our smaller improvements.
Here is possible wording for the proposal that could acknowledge that:
"After each formal usability testing session, the documented findings and recommendations will be delivered to the Drupal Community. Individual developers and small teams will take on different recommendations. They may provide one or more than one proposed solution to the improvement of these forms and features within Drupal. They may also conduct less-formal usability testing and evaluation in order to refine their improvements."
"After 6 months of this Drupal Community effort, the subsequent formal usability testing session will validate all the usability improvements that have been made in response to prior sessions. If there are competing proposals for the redesign of key forms and features (such as the main Drupal Admin page or the Node Add page), this formal usability testing can test two or more of the proposed solutions in order to objectively compare which solutions best meet user needs."
"The formal usability testing proposed will complement the many less-formal usability efforts in the global Drupal Community and leverage the entire community's effort towards the common goal of lower technical barriers to entry for Drupal."
www.pixotech.com
You only need to test with 5 users
Keep in mind this:
http://www.useit.com/alertbox/20000319.html
So with all the effort, there could be run three different task-scenarios with 15 participants or at least two with 14 people if you say 7 per scenario.
Life is a process
Life is a journey, not a destination
cool
given the success of the previous UMN usability testing, this proposal makes sense. D7 can greatly benefit from it.
Formal usability testing
Hi catch,
I believe formal usability testing is very important to improve Drupal. Gave it a 5 for the proposal!
I'm certain many universities besides UMN and UB could help perform these testings. And not a limited one restricted to a few students, but a campus-wide experiment. Most universities that use Drupal would have a direct benefit from this.
I'll try to convince local faculty staff to study Drupal.
Good luck with your project!
Best regards,
Nick
Drupalcon DC
Quick update - since the KDI has shut down for this year, we weren't able to get funding in time for the Drupalcon DC session. However we did get a lab at the University of Baltimore via Becca Scollen - so we'll be going ahead with that round of testing without the funding from Knight. We're hoping to secure some other sponsorship and that some people will be able to attend courtesy of their companies.
I've just posted a panel session proposal for Drupalcon for the testing report back - http://dc2009.drupalcon.org/session/usability-testing-university-baltimore - votes welcome of course.
Solutions in report from Ball State University
Thanks for the great proposal catch. +1
Maybe some solutions presented in this report from Ball State University could help with the formal usability testing of Drupal in 2009?
---
How to contribute to Drupal.