2015-05-12: We discussed this at DrupalCon LA. See update below.
Camp organizers for Twin Cities Drupal Camp are in contact with the University of Minnesota Usability Lab about the possibility of another usability study there in connection with this years camp/sprints!
So far, the lab is available and we have at least one University of MN department willing to "sponsor" our use of the lab. However, we need to fill out a more formal request that outlines our goals for the project.
I started a conversation with yoroy in IRC and he suggested creating this WIKI page to get the discussion going. I'm checking back to get more details on exactly what information we need and when, but to get the ball rolling, I'd just like further input on the following questions.
1) Is this a good time and location (Twin Cities - June 25th - 28th) for a usability study?
2) Who is available to help plan and implement this and potentially participate in a usability sprint at camp? (FYI - We will have sprint rooms available for all 4 days of our camp)
3) What are the current usability priorities?
4) Would this be focused on Drupal 8?
In addition, I can offer that our camp does have some limited funds available to help offset travel expenses for sprint leads. Let's talk more about this.
DrupalCon LA BoF Notes
When?
- June 23/24: Testing at University of Minnesota
- June 25: (crank on slides :))
- June 26/27: TC DrupalCamp (Angie to keynote Friday, Bojhan/Lewis to give wrap-up session on Saturday - session submission deadline May 20)
- June 28: Sprint day (work on fixing stuff we found)
Small travel budget for key participants, but basically this is already taken up. Any other participants need to be local or have sponsorship by their employers/other organizations.
Goal
- Identify smaller issues that we can fix before 8.0.0
- Big, hairy things we should continue to work on for 8.1.x+
- Validating the improvements we’ve made.
Who should we test?
- Web developers
- Some with Drupal experience, some without
- No Drupal 8 experience
Generally, technical users (not necessarily developers) that are directly in Drupal core’s target audience. Need to work with UMN on compensation for participants.
What should we test?
In general, we want to focus on areas of Drupal 8 core that have changed the most or that represent big conceptual things to understand Drupal, as well as those we haven’t tested recently.
- Navigation (Toolbar + IA + menu UI)
- Authoring experience (in-place editing, new content form, previews)
- Content modeling experience (Field UI and Views)
- Block UI
- Mobile* (need to check on capabilities)
Questions for the UMN team
- Can we livestream the usability testing to hand-picked remove observers? How about the whole community? :)
- What tools, if any, support that? (join.me)
- How does recruitment work?
- Can you record testing on a mobile device? Their device or your device?
- What level of promotion is okay to do?
Next steps
- Tim to send out Doodle to schedule call with UMN folks next week.
- Dani, yoroy, Bojhan to start on specific goals, basic outline
- Angie to work with TC DrupalCamp for keynote outline
- Everyone: Come up with a list of people to invite
Comments
This is great news. D8 is in
This is great news. D8 is in the final stretch towards release, it's actually a good time to do a test. With the changed development cycle for subsequent D8 releases this is a good time to find out what works and what needs attention for the 8.1 release.
So, would probably make sense to build off of the script from the previous study. Maybe drop some of the tasks and replace them with new ones that explore new D8 features (preview, views?)
If someone has the old
If someone has the old script, I can update the task flow. I also have the ability to run quick tests on either Verify (with screenshots) and Validately (unmoderated), which could be useful.
Dani Nordin :: the zen kitchen
http://tzk-design.com
http://twitter.com/danigrrl
http://daninordin.com
Thanks
Thanks Dani,
https://www.drupal.org/node/1166656 has the script from the previous session. https://www.drupal.org/node/1162000 lists all previous usability test sessions that we have reports on.
Poifect! I can work with
Poifect! I can work with this. I have a SUPR-Q questionnaire that I can donate to the cause as well, and I can probably round up some Bentley students to do testing in MA. ooh! I could even do some testing in LA. mwahahaha
If someone can handle moderating in Twin Cities, I can get moving on a test plan and post-survey.
Quick questions:
1. Is Jeremy still our main persona for this? Is there anything new we've learned about Drupal users, or anyone new we're targeting?
2. Are there any specific questions about the current state of Drupal 8's interface that we want to test right now that might differ from the original plan?
Dani Nordin :: the zen kitchen
http://tzk-design.com
http://twitter.com/danigrrl
http://daninordin.com
Poifect! I can work with
Poifect! I can work with this. I have a SUPR-Q questionnaire that I can donate to the cause as well, and I can probably round up some Bentley students to do testing in MA. ooh! I could even do some testing in LA. mwahahaha
If someone can handle moderating in Twin Cities, I can get moving on a test plan and post-survey.
Quick questions:
1. Is Jeremy still our main persona for this? Is there anything new we've learned about Drupal users, or anyone new we're targeting?
2. Are there any specific questions about the current state of Drupal 8's interface that we want to test right now that might differ from the original plan?
Dani Nordin :: the zen kitchen
http://tzk-design.com
http://twitter.com/danigrrl
http://daninordin.com
I'm interested!
So happy to see this moving forward. I was part of the original UX study several years back and would love to help again. I would most likely need assistance with travel expenses if I was going to help onsite. I'm also happy to help with updating scripts and identifying new D8 features to test.
Danielle Sheffler
Engagement Manager at Acquia
http://www.acquia.com
Danielle (hi, I'm Dani!) are
Danielle (hi, I'm Dani!) are you local to Bahston? If so, let's coordinate sometime. I'm in Watertown.
Dani Nordin :: the zen kitchen
http://tzk-design.com
http://twitter.com/danigrrl
http://daninordin.com
Hi, Dani! I'm actually down
Hi, Dani! I'm actually down in the DC area. If I can't make it to Twin Cities, I know i can round up plenty of people here for testing!
If you want to work together on the script, any new personas, etc. I can always meet through hangout or Skype.
Danielle Sheffler
Engagement Manager at Acquia
http://www.acquia.com
Also, if I can get travel
Also, if I can get travel expenses covered, I could run a study during the weekend. I just can't do Thursday and Friday.
Dani Nordin :: the zen kitchen
http://tzk-design.com
http://twitter.com/danigrrl
http://daninordin.com
I would hope for a different user group
Hi,
Site builders user interface of course always needs to be developed, but I personally would find it useful to focus on content contributors.
I guess this depends a lot on one's own perspective. I have developed sites for NGOs and academic groups with Drupal since 2007. I myself find it useful to have already experienced people building a site but there always needs to be an easy enough interface for the content contributors. Should there always be a need to redesign the content contributor interface to be simple enough would, I guess, scare away new site builders as well.
Maybe one good goal could be to try to identify a good set of default user roles and permissions defaults. What needs to be shown always, what can be more hidden, and what is never needed. And when something is not immediately visible, where people would like to find it.
But this is just my idea, based on how I usually work.
Visitor user group then of course tells more about the site design than Drupal. One such site study was conducted at Helsinki university in 2013 as part of a usability testing methods seminar. We tested the computer science department's Drupal 6 site using 7-8 different methods. (I did the traditional usability testing). I guess the only Drupal specific finding was usability problems with search results.
I don't see why we couldn't
I don't see why we couldn't do both. We'd have to create two test plans, but ideally we'd want to uncover the needs of both groups.
Dani Nordin :: the zen kitchen
http://tzk-design.com
http://twitter.com/danigrrl
http://daninordin.com
I don't see why we couldn't
I don't see why we couldn't do both. We'd have to create two test plans, but ideally we'd want to uncover the needs of both groups.
Dani Nordin :: the zen kitchen
http://tzk-design.com
http://twitter.com/danigrrl
http://daninordin.com
Fantastic!!
Thanks so much for this exciting initiative! I will begin running it through my 3 levels of approval post-haste. :) (boss, wife, childcare) If it's a yes, my employer should be able to fund my travel, so I would not need sponsorship.
1) I think any time the U of M is willing to provide their awesome usability lab is a great time for a usability study! :D So, yes!
2) Count me in for helping in whatever way I can: planning, participating, writing up/presenting the results, etc. I've attended two prior usability testing sessions at University of Minnesota (which has been an amazing supporter of the Drupal project!) and assuming it's the same set up as last time, I'm comfortable playing any of the various roles (scribe, help desk, eye-tracker-watcher, etc.)
4) Yes, definitely Drupal 8. As yoroy mentioned, the end of June (which should be somewhere around RC) will be a great time for this type of testing, as we can queue things up for Drupal 8.1.x and beyond.
3) I would say we should focus on anything new in Drupal 8. https://www.drupal.org/drupal-8.0 That would be things like:
1) Content authoring (redesigned node form, preview, in-place editing, toolbar)
2) Mobile support (doing content authoring/editing + site building tasks from an iPad/phone)
3) Various site builder stuff (Views in core, new block placement UI, new fieldable entities like custom block types and contact forms, new field types like date/entity reference)
4) Configuration management (also probably not a good fit given the set up required [need two sites, easy to get them confused], though it would be interesting to see what kinds of hilarity ensues)
5) Multilingual (probably not a good fit given the amount of set-up required and the fact that it's only about 15% of installs)
6) Web services (not an easy way to UX test this unless we want to analyze how quick people are able to do e.g. an angular front-end with Drupal, but that'd be pretty boring to watch. :))
7) HTML5/Twig (same; coding stuff.)
8) OO code (once again coding stuff, but could time how long it takes for a non-developer vs. a developer to create an example module; might be interesting.)
I definitely have a preference for testing things with the most "impact," which would be the first 3 (possibly 4), IMO in that order. (There's only ever one user 1 on a given Drupal site, but there are N or NN or NNN content editors.)
d8mi
since you bring up multilingual, note we had some usability testing in 2012 and 2013 focused on multilingual (which contains scripts) https://groups.drupal.org/node/271918
Cathy Theys
Next steps?
Hi, Tim.
Is there a template or something we could jointly fill out to send the request? I'm a bit worried that 6/25 is not too far in the future, esp. if we need to wrangle sponsorship money.
Update on Next Steps
I've been communicating with Bojhan about logistics via email and a Skype call today. The current situation is that we have the lab scheduled for Mon-Tues (June 22-23rd). This is a bit early, given that camp doesn't start until the 25th (Thurs) and we're hoping that folks will stick around for camp.
The guys that work in the lab are at a conference on the 24th and 25th. We're checking to see if there is any availability on either of those days or Friday (26th).
TC DrupalCamp has some funds for limited travel expenses. We're willing to help other guests find accomodations in private homes. I don't think we need any funds for the facilities. So, the only additional sponsorship money that would be required is for additional travel/accomodations that might be required.
We're also willing to see if we can come up with a little more funds from our budget.
I am still trying to figure out if there is anything else the U needs from us before next week. I'll be in LA and can coordinate with Bojhan and others in LA.
Tim Erickson
Triplo
Honestly, a couple of days before the camp is great.
Honestly, a couple of days before the camp is great. Better even. Because from past experience, the data coming out of this is going to be fast and furious, and we're going to need to do some work sifting/compiling in order to make it useful/understandable to the wider community. It sounds like this would give us the entire day of the 24th to get some slides together, and then possibly present those on the 25th to kick off the camp?
+1 for meeting up at LA and talking more though. :)
Great
Thanks, that's great feedback.
To clarify, camp starts on Thursday the 25th, but that would be training day. Presentations and sessions are the 26th and 27th and we have a dedicated sprint day on the 28th. We'll have sprint rooms available all 4 days.
We're afraid that if the usability study is on Mon 22nd and Tues 23rd and sessions don't begin at camp until Friday the 26th, that this would mean an entire week of travel for someone from out of town. We'd like for folks to be able to participate in both the study and still be around for camp.
Tim Erickson
Triplo
While we're in LA, let's talk
While we're in LA, let's talk about how much of this might be able to be handled remotely as well. I definitely can't travel, but I do an extensive amount of remote usability testing at work, and could easily help with analysis and moderating from the Boston area. Maybe there's a way for some work to happen in MN and some to happen via conference call/screensharing.
Dani Nordin :: the zen kitchen
http://tzk-design.com
http://twitter.com/danigrrl
http://daninordin.com
Good news - False alarm
It turns availability of the lab is a little more flexible than I expected. We can have whichever days that we prefer the week of June 22-26th. I've leaving that decision to folks like Bojhan and Webchick (or others who might be coming) to make a decision about which days will work best for participants travel schedules.
We just need to decide by next week. We can make this decision as late as Tuesday of next week, when we meet in LA.
They don't need anything else from us right now. They will be available for a short conference call either during our meeting next week in LA or sometime the following week.
Tim Erickson
Triplo
Exactly, would be great if
Exactly, would be great if you could provide conference call/screensharing. I would like to watch and help but im from Poland (Europe).
We met about this at DrupalCon LA
We (Bojhan, timonweb, wylbur, lewisnyman, danigrrl, webchick, yoroy) met at DrupalCon LA to talk about what to test, etc. BoF notes in the summary. https://groups.drupal.org/node/467458#drupalcon
webchick, this is great! I
webchick, this is great! I probably can't come since I'll be taking some time off for Drupal GovCon in July. Is there any way that while usability testing is going on at TC, that I could run some testing here in DC? I'd love to be involved in some way!
Danielle Sheffler
Engagement Manager at Acquia
http://www.acquia.com