GSoC Project Idea: Usability Test Suite

Added to official ideas list at http://drupal.org/node/234746

GSoC Project idea: Suggested and discussed at UMN after Usability Testing as a way of Measuring the UX (user experience), and further discussed with Jimmy Berry, who is interested in taking this on as a student. Measuring the UX is the primary UX goal for Drupal.

This needs more work, but is a starting point for a project proposal.

Glossary:

• 'data' refers to user-input text and feedback, logs, video and audio
• 'usability unit test' is misleading. I want to capture the concept of the same scenario, tasks, and interview questions that multiple evaluators are given.

Deliverables:

• Usability Unit Test module
  • allows a test designer to create a Usability Unit Test, including
    • evaluator requirements questionnaire (to validate evaluator's eligibility)
    • privacy statement / agreement
    • evaluator demographics, background and experience questionnaire
    • scenario document
    • tasks, at least one per test
    • for each task
    • list of actions that indicate success or failure in completing the task.
      • include php snippets and/or actions/triggers to identify when they occur
      • e.g. 'Created new content type', 'disabled content_access permission for anonymous role'.
    • task feedback questionnaire
    • optional dynamic elements to questions
      • e.g. only ask 'What made you click the link to FOO?' if the user clicked that link.
    • test feedback questionnaire
    • drupal data capturing requirements
    • object types to log/watch
    • database dumps
  • for each test/evaluator, coordinates
    • several data-capturing modules and/or browser extensions
    • elements of a usability test, such as
    • questionnaires, agreements
    • statement of purpose (software is being tested, not user)
    • scenario
    • tasks
    • feedback
    • anonymizing data, if necessary
    • collection of data from multiple sources in a central repository
    • support for S3 (existing module) would be valuable here, since raw data is large
    • post-processing of captured data
  • generates 'across-the-board' data such as
    • heat maps
    • analysis of user feedback and logs for common and repeated keywords, success points, failure points, incomplete tasks, etc
    • output
• Modules and/or FF extensions to Capture and Store:
  • mouse movement
  • clicks (Click heat map module)
  • drupal log of
    • clicks / uri's / paths / pages
    • form submissions
    • changes to drupal objects (nodes, node types, taxonomy, terms, blocks etc.)
  • audio -- user's thoughts
    • user text log as a fallback if no microphone or other technical issues
  • screencast
  • videocam
  • database snapshots

Everything needs synchronized timestamps -- use the same source for timestamps. Since some of these data capturers may not know about the web server where other data capturers send their data, we might need the server to use client's timestamp instead of local, or vice versa.

The accruacy of the time is unimportant, nor is the timezone. Only the relative times of all of the data is important, in order to post-processing and reviewing data. To facilitate this a consistent time format should also be used, with granularity smaller than 1 second.

Bundle and compress merged, and raw data and save locally, or send elsewhere (email, sftp?)

Server can intellegently merge video, audio, logs and other data into easily-portable and viewable file/s -- such as one combined video file with data split screen and data overlays.

For example: If data has been captured from all of the above sources, create video with split screen of screencast and videocam. uri or path and summarized drupal log are overlaid like subtitles. Audio is included.

If no audio was available, overlay the user log.

If no screencast was available, overlay mouse movement data on screenshots.