Posted by dcmistry on February 28, 2011 at 7:38pm
Hi All,
I am considering to present a talk on open source and usability. Since the scope of this is too broad, I am focusing on the BIG CONCERNS of open source (especially Drupal) with respect to usability and user experience and the ways we intend to tackle them. I am particularly interested in getting YOUR reactions to this as it provides me with a fresh perspective on things.
Thank you!
Comments
Check out bojhan's talk on UX
Check out bojhan's talk on UX design in open source: http://www.slideshare.net/Bojhan/design-in-opensource and http://www.slideshare.net/Bojhan/user-experience-in-the-drupal-universe
Your question is still very broad! :) Are you looking for process, tactics, strategic challenges? Big concerns in a nutshell: hard to define, scope and STICK to a design vision. Hard to manage distributed efforts. Ratio in Drupal is like 1 ux designer for 100 core developers. Having an actual design process: educate on design not being a taste thing but that there are design rules, derived from how humans work (cognitive psychology, hci etc, you know about that better than me).
Happy to talk more but that's it for now. Nice to see you join the discussion here!
It would help to hear details on this in the issue queues
"This" being:
Especially when a conclusion about the usability of a proposed interface conflicts with my own understanding of these principles and testing of human behavior on the Web.
Instead, all we usually get is a vague mention of aesthetics as being the reason a patch proven to work can't be implemented.
There are many more of us UX-ers waiting in the wings. It's just hard to participate fruitfully when "aesthetics," not a clearly referenced set of principles, are used as the rationale for a seemingly arbitrary decision.
As one bright and clear example, the case of the password checker comes to mind. Many developers participated in good faith on that project just to have it blocked at the end by "aesthetics," and clearly stated requests for explicit options for addressing that issue went unanswered.
We UX-ers are here, Roy. But when discussion is cut off like that, we can't help out.
I'm really sorry that I can't be at DrupalCon to discuss these issues in person, but everything I contribute to Drupal is done in my spare time, often in the hours after my family is asleep. I can't afford the expense, nor can I devote my vacation time to this conference. If I could at all be there, I would.
Review of Bojhan's slides
Just a few points:
A problem we've had, Roy, is that designers have tried to do their own usability testing. They've drawn conclusions that were far too broad, and, as far as I can tell, they did no testing of early prototypes of potential solutions to the problems they found. How do we know they found the simplest and best fix?
We usability experts are here. And, if there could be a "Professional Societies and Associations" field in our user profiles, we would be a lot easier to find and organize when our help is needed.
Back on subject...
I think we desperately need to discuss strategies for addressing user experience in an open source project. Please keep the discussion going!
My thoughts
Thank you Roy and Cliff for the response.
Roy - Yep, still very vague and broad. Chewing more about this, I was thinking more on the lines of the process we follow. I think, there is a good overlap between process and strategic challenges as the challenges shape the process. The interesting thing is that the challenges keep changing but the process is hard to change.
I completely agree that there is a need for awareness when it comes to design. Since I am not a pro here (unlike you :) ), can you point out what has worked well and what has not (wrt to educating the other members in the past)? May be we should ruminate on some ideas.
BTW, Bojhan's slides are great!
Cliff - No worries, if you did not get to attend this DrupalCon. I could not attend either :)
You make a valid point about the conflict between principles and behavior. I see the "principles" as guidelines. Some of these principles/guidelines are helpful while some are not. It certainly depends on the target population - who is using and what are they using it for.
I am not aware of the password checker instance but I do support that we need to look at other factors like functionality, usability and experience along with "aesthetics". Aesthetics is an important one, but not the first and certainly not the only one.
Dharmesh Mistry
UX Researcher | Acquia,Inc.
2 cents
I can think of a few issues.
A UX issue always has a corresponding code issue, you can't solve the UX issue without a developer writing a patch. Likewise you might have a coding issue (for example adding a feature) that has a corresponding UX issue.
There are a number of issues hiding within this dichotomy:
We're told to start with design then write a patch, but what if a designer is not around? Do we hold up development waiting for a designer to turn up? No we don't, we get on with solving the problem. This is a big problem - developers need to be more UX aware and wait, even hunt down designers and UX people to get them involved early in the process, likewise the designers need to be better organized, more involved and I would argue better at picking their battles (we can't fix everything, seasoned Drupal developers know this and pick battles carefully).
At what point is an issue fixed? This may well differ depending on which crowd you're part of. UX might sign the patch off as soon as the design objectives are met, in fact this happened quite a few times during Drupal 7 development. However the code may be far from satisfactory. Vice versa happens also, code changes are approved yet the UX objectives have not been met. This is a problem - we need a better system for approving patches. We did have a kind of ad hoc "needs UX review" thing going, but this is weak, we need a better system.
Pretty much what Cliff is alluding to. UX people need to back up their arguments with evidence or a really solid case. All to often developers were fobbed off with weak aesthetic explanations, even worse often treated as "stupid" when asking for more detail. When developers write patches or make feature requests we have to really argue hard why something should happen and often write code as POC. UX teams need to take a leaf out of the developers book and get their hands dirty and do design, do make solid arguments, do cite design principles and explain in fine detail why and how these apply. We have to do with with code, UX should have to do it with design.
So I have raised what I think are 3 important issues:
1 - if UX wants better UX, get involved.
2 - We need better processes.
3 - Use better arguments.
Strongly agree with #3. To
Strongly agree with #3. To give a very simplified view of this (I could use a more real example but I don't want to be argumentative):
Coder: "Ok here's the patch that adds the foobar widget."
Designer: "This looks horrible. It really kills the overall feel and UI experience."
Coder: "Ok, what is the problem exactly with it, and what would be the suggested change."
Designer: "I don't know it just doesn't feel right. It needs to have herp and derp more."
Coder: "What are we trying to accomplish here?"
Whereas things would have gone much better if the conversation more like:
Coder: "Ok here's the patch that adds the foobar widget."
Designer: "Nice work, but the overlay element uses #FFFFFF whereas #FFEEFF" would blend in better here (See xx page for why this color goes well that color). Also, if you could move the derp element up to the top of the widget, the user would have a more logical path and the herp would really stand out more."
Coder: "Ok, here is a re-rolled patch, thanks for the review."
Anyways, that's just my perspective on it.
Drupal evangelist.
www.CoderintheRye.com
Deconstructing Babel
I can see how there will be lots of friction between designers and developers (as there has been since there have been humans, probably) if they do not understand each others' motivations and if they cannot walk a mile in the others' shoes, so to speak. The more understanding we can foster between the two camps, the better the end result will be. This sort of thing is enforced in companies, but harder to come by in a wholly self-organizing system. Bojhan elucidates this fairly well in the above-referenced slides. There is clearly a cultural gap that needs closing, and when the two sides feel at odds, it makes for a hard time.
It would be useful to develop a set of Drupal Design Heuristics to use as a starting point for negotiations over UX and patches. I believe I've seen something along those lines before, but can't find the page presently. Can anyone point to it (if I'm not imaging things) - I see this from Roy and Bojhan: http://www.yoroy.com/2009/drupal-user-interface-design-pattern-library . Basically, a set of general guidelines to not only help steer contributed modules and core stuff into a consistent UX state, but also to explain why certain design elements are un/desirable - beyond a hand-waving over the issue as aesthetically unpleasing.
I'm hesitant to point to a standards body for an example, but the document here: http://www.w3.org/TR/WCAG20/ shows how to state a principle, derive guidelines that illuminate and support the principle, and further provide examples for implementation. The W3C standards format is a little Byzantine, and I would imagine something useful would provide a friendlier format. Supporting evidence for each guideline would be a good addition - so a designer can point to the standard behind his/her suggestions. Once a standard like that becomes a part of the culture, people will hopefully have to do less explaining, on the one hand, and in the spirit of "choosing battles", a standard can make implicit which sorts of things are worth arguing over.
Sorry if I'm retreading old terrain; I'm new here.
...
Yes, that is precisely what I am after - something UX can point at and say this is why. The "because I said so" argument only goes so far, we need standards that are transparent and which help distill the most important battles.
Are you talking on guidelines
Are you talking on guidelines like this one - One of the usability heuristic evaluations which is widely used in the industry is by Jacob Nielsen http://www.useit.com/papers/heuristic/heuristic_list.html . Although this one is a great example, a lot of times issues come up which do not fall into one of these categories but are still important. In such cases (in fact in all cases), a clear rationale should be provided.
On a side note, I was glad to hear Dries talk a lot about usability and accessibility for D8. I think such discussions are helpful to iron things out and have a better process.
Dharmesh Mistry
UX Researcher | Acquia,Inc.
That, and then some...
These are classic usability guidelines, and are a good starting point, but they are still subject to much personal interpretation and are not formalized/operationalized enough to overcome the problem we're talking about. In a couple of the books I've written on web and mobile device usability, I have some examples of fleshed-out guidelines, but even those are lacking the citations for the research behind the decision tree.
I'll see if I can track down the discussion thread mentioned here about the password dialog and see if I can make an example entry in a design standard, one that shows the principle, the guideline, the rationale, and the example for implementation.
Regarding accessibility and usability, I think they are symbiotic. A lot of the rationale behind accessibility in design was initially driven by ADA and Section 508, but it turns out that accessible web sites have the side effect of being decent on mobile devices and other alternative browsing methods.
I'm assuming that this is the thread...
http://drupal.org/node/331893
... where everyone was debating how to implement a password strength checking UI element?
Looks like a good example case of what we're discussing. Let me see if I can come up with an example set of P/G/R/E that could be used in such a case. There are several universal threads in the discussion - use of text versus visual cues (color), meaning of color, visual metaphors, etc.
The one with 98 comments? Bingo!
For what it's worth, I consider it the classic example of what I was referring to above.
Be careful of citing heuristics...
Before we get too far down the "list of heuristics" road, we need to bear in mind the work of Rolf Mölich. Mölich worked with Jakob Nielson to develop the original set of heuristics.
Problem is, even when applied by teams of well-respected, highly experienced researchers, the heuristics don't lead to a consistent set of answers. Critical problems were missed, and there was very little overlap even on the less significant problems.
Furthermore, even when usability testing was done, most of the critical problems were missed by all teams and there was very little overlap on the other problems.
Robert Hoekman Jr. has written an article summarizing Mölich's findings. You can find it on A List Apart: The Myth of Usability Testing. He has also done an outstanding lightning talk on this topic; if I can find that link, I will revise this comment to add it.
The whole point is that, as useful as usability guidelines and even usability testing are, there is still an element of expertise involved. Experience does matter. And not the "I read a book, watched a webinar (or set of usability tests), and tested a half-dozen people myself" kind of experience, either.
The more experience you have, the more you understand that there are some general rules that are constant — except when they aren't.
And the trick is in knowing which the conditions you're reviewing are more likely to be like — the most common outcome, or the exception?
Research-based design link, and another note of caution
I've not really been following the UX discussions in D7 and D8, aside from some of the items specifically related to accessibility and the WCAG. But I'm happy to point out what I think of as one of the best, freely available, "research-based" web design resources, the Research-Based Web Design and Usability Guidelines produced by the U.S. government:
http://www.usability.gov/guidelines/
Amongst various goodies in this document, you can find a page discussing the issue that Cliff raises about whether or not "heuristic" analysis is actually very useful in the design process (see section 18.8 Use Inspection Evaluation Results Cautiously on page 194, where they claim "Inspection evaluations should be used cautiously because several studies have shown that they appear to detect far more potential problems than actually exist, and they also tend to miss some real problems. On average, for every hit there will be about 1.3 false positives and .5 misses").
I like the idea of Drupal having some basic agreed-upon guidelines relating to design/aesthetics/processes as they relate to user experience and accessibility. But I think there is a great, almost insurmountable, challenge facing anyone trying to find strong, objective "evidence" to support many of the kinds of specific design issues that would arise. Coders will argue, sometimes vehemently, about different coding solutions, but there are a handful of evaluative principles that one can call on to compare solutions. For example, one might appeal to principles like efficiency (whether processing speed or number of lines) or readability or modularity, and one can also always appeal to the question of which solution actually works by delivering the correct result on the page. By contrast, web design examples have very few agreed upon principles, I think.
When I first found the usability.gov Guidelines book, I thought everything would be easy in terms of design choices from then on, because I would have a basis from which to work. But what I quickly discovered was that many of the "guidelines" in the book are actually supported only weakly by evidence. They try to find the best evidence they can, but there just isn't good evidence about a lot of it. Part of the reason for this is that most usability studies are actually the result of paid consulting jobs and the product is never made public: there are few "open source" usability studies. Nielsen's group has released perhaps more data than almost anyone, but most of his data is still strictly confidential I think. Also, it is very difficult to create a good usability test around a design principle. On the few occasions when I have had occasion to actually read through a full set of results of a usability test, I am almost always disappointed by what I see as clear flaws in the design of the test. In the context of the development of such tests, the web is still very young, and I would say the evidence around almost any web design principle you care to look at are still very much contested ground.
For instance, even though I'm not a researcher, I personally disagree with some of the "principles" that Nielsen expounds from time to time -- as do other designers. I even disagree with some of the principles listed in the Research-Based Guidelines that I've just finished holding up as the best example out there. And, sad to say, my disagreements are based on personal conclusions (i.e. experience or whatever), not strong evidence: I may be able to cite a specific example of a case that demonstrates where my disagreement comes from, but I know that that is just anecdotal evidence, and it is the kind of evidence that wouldn't convince me if I were arguing the other side.
And I'm someone who has spent time trying to look for "research-based" evidence to support some design choices. I know designers who are vastly superior to me, who have an ability to develop a UX design that I immediately, almost instinctively know is better than whatever it is that I am doing, but who really don't rely on research evidence at all, and who may not be at all good at justifying their work on that basis. User-testing has a strong basis in web design, but I would say that it is very rare to see good user tests actually deployed: designers are usually left with directives from their clients that make very little sense from a UX standpoint, and they then have to figure out ways of convincing their clients to accept other arguments -- including "experience" -- even though in an ideal world, a perfect set of user testing results would iunstead supply a designer with the perfect information needed.
The goal of producing a set of clear, usability design guidelines for Drupal is a great idea, and one that I support, but I would guess that it is gonna be a long, tough haul, and even then, I bet we'll still end up having some of the exact same kinds of debates that the existence of such guidelines would hope to prevent.
Phil.
Doh! Why didn't I think of that?
Perhaps the best feature of the Research-Based Usability Guidelines is that each one is rated according to importance of the guideline and weight of experimental evidence supporting it. As you point out, some things that are deemed important have little evidence to support them. And some things that are well supported by evidence are just not worth stopping a patch over.
On the (many) issues these guidelines address, they either give you a clearcut answer or help you have a relatively informed discussion. And if they lead you into a discussion, they also give you a pretty good idea of what's at stake. That's good information to have.
They're a great model. And we could do far worse than to adopt them by reference.
Phil, thank you for chiming in. Great post!
Agreed
The usability.gov guidelines are an example of what I'm talking about, too, and I agree that we could do far worse than use them as a starting point for a set of Drupal-specific usability guidelines.
I also agree that so-called "heuristic evaluations" are not all they are cracked up to be; putting actual users to the tasks is the only empirical way to see what holds water and what doesn't. Like any tool, they are useful to an extent, but only when used in a cocktail of other techniques. You can't build a house with only a hammer.
Most outfits that perform usability audits are indeed constrained by either non-disclosure agreements or by the various parties' desire to maintain an edge. Out of the scores of usability clients I've had, many of them large banks, none of them would ever agree to allow me to publish findings specific enough to be of use in the "open source" sense. And the big usability firms often like to keep their trade as arcane as possible for self-preservation.
Phil, you're probably right that generating a community standard for usability in Drupal will be difficult and that it won't solve all the possible problems. This is not to say that it wouldn't be worthwhile, though - unfortunately only time would tell. But it seems to me like if this group is interested in improving usability in Drupal, well, that act itself will also be difficult and won't solve all possible problems. Here's an opportunity for the Drupal community to take a leadership role in open-source usability...
Patterns
I found an interesting post: http://angrylittletree.com/11/01/drupal-8-road-ahead
In that discussion eaton talks about UX, I thought this particular snippet is pertinent:
That would be a great start.
Two Words: Karen McGrane
For people who are wondering what you're talking about, view Karen McGrane's excellent presentation from Do It with Drupal in 2008, Creating Usable Websites with Interaction Design Patterns. Karen is at DrupalCon right now, I believe, and I plan to pick her brain on this very topic when she returns to Austin, where I'll run into her at South by Southwest Interactive.
For those who can't see, skip to the transcript, which is structured as a numbered list, and listen to these slides:
You won't get absolutely everything, but you will get her main points. (Warning: Each slide ends with "Do It with Drupal 2008," so skip to the next when you start to hear that.)
As with accessibility, this is a difficult area. Some of the control is in core. On any given site, some of that control will be in one or more modules. And then, of course, themes can override at least some, if not all, design features from modules and core.
But even if all we give people is the idea that a particular pattern is a good target, that would help improve usability. If we could add one or more ways to hit that target and have the site be accessible, that would be even better.
http://www.slideshare.net/Boj
http://www.slideshare.net/Bojhan/drupal-7-interface-pattern slides from our session during Drupalcon Chicago.
And: http://drupal.org/ui-standards is where we want move those docs. We've been prototyping it on http://p1.drupalusability.org/
Just a quick post while attending a great core conversation, great conversation here, thank you.
Leading research comes from
Leading research comes from military and gaming circles. Useful info can be had from heat maps, which give info based on what part of the screen the user is focused on - too expensive, but there's some general lessons that can be learned from what's on the internet. There are terms such as above the fold and below the fold, which come from seo, which is useful for advertisers. If you go to last.fm, you see one example of out of many, which base operations off of subscribers. UX considerations would be broader than UI design, and would include how fast to load the page, among other business topics. The links are lacking in presenting enough info which would bridge the divide between business goals and UI solutions to meet those goals. Non-profits are better served by thinking about lessons from business for their operations. Navigation, Inputs, Notifications, Dashboards, Checkout/Form pages, are some categories useful to breaking down this discussion into smaller groups of problems. Module categories for drupal can provide additional ideas, with the criteria being the category in question presents an UI.
Patters
I was very much reminded of this demo when looking at your prototype:
http://hanshillen.github.com/aegisdemo/
Possibly because there are some common elements, but also because there might be some common code.
--
OpenConcept | Twitter @mgifford | Drupal Security Guide
Future directions
Although slightly off-topic, but maybe the Drupal core team should participate in this EU tender?
http://ted.europa.eu/udl?uri=TED:NOTICE:76089-2011:TEXT:EN:HTML
My two cents
I am glad that people are engaging actively on this one. This discussion is going to help a lot for D8 and listening to Dries at DrupalCon, it re-emphasis the fact the usability and accessibility needs more attention than what it is.
To some of your points earlier:
Cliff - Certainly the results from CUE studies do raise one's eyebrows when it comes to heuristic evaluations. In fact, I did participate in one of those (CUE 8). It is always a hot topic discussion in the community. I would suggest to think on the other side as well which is although the results are inconsistent, some of the study protocol implemented are different (like the recruiting screener, the participant demographic, the moderator's effect, etc). That being said I think the later CUE studies had a more standardized approach to the tasks but still a few things are always a variable which influence the results. This is always going to be the case. What can we do about it? My take is that heuristic evaluations can be a great technique to implement. However, it depends (we usability people love to say "it depends") with what we are dealing with. Usability testing would be the ideal case (A lot of research suggests that testing 5-6 participants uncovers around 70% of the issues). However, it is not always possible to do usability testing - it is time consuming and it costs money :) For some features, we can possible give heuristic evaluations it's due worth and try it out. It requires technique and expertise on usability front and knowledge about users and Drupal.
To vividgates's point about eye tracking. It is certainly a fascinating technique. It adds a lot of visibility and influence. As you pointed out it is costly. But we can reserve them for HOT features. Perhaps collaborate with labs which has eye tracking equipment? For other features which could not justify the cost of eye tracking, we can try the probing technique. It is not 100% accurate but a skilled moderator can yield substantial results.
Going back to the general theme of guidelines: I agree it is hard process. No one guideline will fit perfectly to our needs. We just need experiment a bit and see how it goes. This may be a good place to start or may be we need to scratch a bit more. I prefer scratching more now and then see how it goes. I have my set of concerns of trying to implement guidelines (like most of you) and if it is the optimum solution.
To conclude, what I am doing and will continue to do is keep doing usability studies for Drupal focusing on various aspects of it and share the results with the community. I intend to use different techniques depending on the need of the hour (with your suggestions, of course) which study fits the best technique. That way we can try to address issues with the bigger themes. For the smaller ones, we need to find a way out. Also, I think doing some accessibility usability testing could be great.
Dharmesh Mistry
UX Researcher | Acquia,Inc.
Upcoming D7 Usability Testing
Just because I don't see it mentioned here, thought I'd point out that there is Drupal 7 usability testing planned for May 17-19 at the University of Minnesota. Honestly I'm not very up-to-speed on the details. But I've been to the lab at the U of MN, seen their eye tracker and other equipment, and it's quite impressive. I also know that Drupal 6 usability testing was conducted there a couple of years ago, and there's a group setup for organizing this round of usability testing.