Green by 2014: Let's get to zero automated accessibility test failures in the Accessibility module

jessebeach's picture

Executive summary

We want an accessibility testing green board for Drupal. Currently, the testing platform and tests are under development and progress is solid, thanks in major part to Kevin Miller (kevee). The tests register 257 failures currently, although some might be false positives and duplicates. We need developers, testers and reviewers to get us to robust test coverage and zero failed tests against Drupal 8 by 2014.

We’re calling this effort Green by 2014.

Please tag all issues with accessibility and Green by 2014.

Current efforts to improve accessibility in Drupal 8

The Drupal 8 development cycle has included concerted effort to improve the accessibility of administration features and content. But we’re not satisfied by uneven improvements. We want Drupal 8 to be the most accessible content management system in the world.

The accessibility module describes itself as “a common framework for checking accessibility of a webpage". It provides facilities to check content and code against standard accessibility guidelines.

This module and its tests are still under active development. We need developers to assist us in bringing these efforts to conclusion within the Drupal 8 development timeframe so that we can declare with conviction and evidence that Drupal 8 meets Section 508 and WCAG AA standards.

Update the Accessibility module code to be compatible with the latest D8 HEAD APIs

Drupal 8 HEAD, although largely stable, still undergoes occasional pattern and API changes. Given this reality, we need to spend time updating the Accessibility module’s 8.x branch to keep up with these changes. Our effort to get to a automated testing green board must start with updating the Accessibility module so that testers can run it locally.

We need individuals who are familiar with Drupal 8 module development in order to advance development of the accessibility module. These folks will:

  1. Create issues in the the Accessibility module issue queue.
  2. If you can, propose patches to resolve them.
  3. Patch reviewers. Please follow issues that you can contribute to!
  4. Please tag your issues with Green by 2014.

Making Drupal more accessible

The Accessibility module helps us identify where Drupal’s access support falls below that of Section 508 and WCAG standards.

This testing tool runs many tens of tests across numerous pages of a Drupal installation. You can either run the tool locally through the Accessibility module or view the daily run at Drupal Accessibility Status (beta).

The tool is not perfect yet, though. Some tests return false positives, meaning that an identified issue is not really an issue. In this case, we need to adjust the test to remove the false positive. For legitimate test failures, we must log an issue against Drupal core and propose a patch to fix the issue.

Logging accessibility issues against Drupal core

In cases of an actual accessibility issue, we must first create an issue to describe it and then address that issue with a patch. The Accessibility module reports issues in a table layout. The test name, theme hook, theme item (function or template) and and output markup are provided. I’ve given one example here in a list format.

  • Test: Text has appropriate contrast
  • Theme hook: page
  • Theme item: core/themes/bartik/templates/page.html.twig
  • Element:

mgifford gives us a great example of a focused issue that arose from the automated tests and anandps provides a patch to review. finds two new color contrast issues.

In this issue, mgifford gives us a reference to the failed test, the specific code that’s failing and a possible path for a solution.

Text has appropriate contrast
page: core/themes/bartik/templates/page.html.twig

<div class=\section clearfix\></div>
<div id=\name-and-slogan\></div>

Does have color contrast problems according to this new Chrome Color Contrast plugin.

In your issues, you are not expected to provide a solution. Simply logging the issues gives us a great start to addressing it.

Drupal already has great issue documenting guidelines in how to create a good issue report. Please follow those guidelines when creating a report.

Please tag your issues with accessibility and Green by 2014.

False positives

The automated testing tool might report an error that occurs becuase of a poorly written test, not because we have an issue with Drupal core code. This is a false positive failure. Most likely the parameters of the test need to be adjusted. You should always first consider, when addressing a test failure, if the test might be at fault if you find in your investigation that the Drupal output passes your manual testing.

False positives must be fixed in the QUAIL project directly on Github. To address them, take the following steps (as far as you are able).

  1. Create an issue in the QUAIL project issue queue.
  2. Tag the issue false positive and Green by 2014.
  3. If you can, propose a patch to resolve it.

Documenting our coverage

It’s well and good to have great test coverage, but we need to communicate that coverage to the outside world. Specifically if we are to make claims of conformance, we must be very sure that our tests convey conformance.

That requires us to correlate our existing tests to sections of the specification and explain how the test satisfies that section. The current state of correlations is documented in the Test guideline alignment.

Send a pull request to update any of the documentation in the QUAIL github repo.

Verification tools

You might try the following browser tools, among numerous others, when verifying accessibility tests.

Wave plugin for Firefox
Color Checker for Firefox
Chrome color contrast tool

Please help out!

We intend to make Drupal 8 the most accessible content authoring and management tool ever. And we intend to make that claim verifiable in part through robust and wide automated testing coverage. We need your skills, dedication and compassion to make it a reality!


Creating new tests against the WCAG 2.0 techniques

jessebeach's picture

Does it make sense to make a mapping these techniques to the tests we have in QUAIL?

Some of techniques are outdated e.g. H24: Providing text alternatives for the area elements of image maps and some are overlay broad e.g. G192: Fully conforming to specifications. I think we can prioritize those very low.

The set of techniques around are a high priority for a CMS.

If you'd like to write a test against a WCAG 2.0 technique, start an issue in the QUAIL issue queue and link to the technique. Every technique's details page has a Test header that describes how to structure a test and the success criteria for the test.

Yes, absolutely - for educational purposes if nothing else!

anandkp's picture

I feel that updating quail.js to map it's tests to the techniques they're implementing would be AWESOME!

It would help serve two purposes:

  1. It would help educate users learn how to write a test for a technique.
  2. As you said, there are some techniques that are out-dated and/or redundant. It would help ensure that such techniques+tests are removed and/or avoided.

@jessebeach, you're proposing something like this?...


aMustContainText : function() {
  quail.html.find('a').each(function() {
    if(!quail.containsReadableText($(this), true) &&
       !(($(this).attr('name') || $(this).attr('id')) && !$(this).attr('href'))) {
      quail.testFails('aMustContainText', $(this));

To something like

 * @see H30: Providing link text that describes 
 * the purpose of a link for anchor elements,
aMustContainText : function() {
  quail.html.find('a').each(function() {
    if(!quail.containsReadableText($(this), true) &&
       !(($(this).attr('name') || $(this).attr('id')) && !$(this).attr('href'))) {
      quail.testFails('aMustContainText', $(this));
stephaniemiller664's picture

Some of details is out of date..You can look in see that every details are some of techniques.

This mapping would be really

Hanno's picture

This mapping would be really great as we can give users additional information. It would be better to add the technique-id in the json file as a id. It would then possible to create automatic links, overviews and statistics.

So, I'm envisioning a set of

jessebeach's picture

So, I'm envisioning a set of UIs for the Accessibility module that pivot around the specifications. Maybe a wall of checkboxes for each item of the guideline that has a test, with config options for a test if appropriate. The form could have a couple pre-baked combos like WCAG A compliance and AA compliance. Or "content creation" tests or "site navigation" tests. Maybe the QUAIL library could expose these configurations to the Accessibility module rather than the module knowing about how to construct these combos. It would make QUAIL more useful to non-Drupal implementations.

What would a sample guideline item look like in configuration?

id: [autogen]
description: 'some text'
type: [technique, normative]
guideline: [wcag, 508, ...]
gid: [G1, H14, 1.14.2, ...]
uri: [uri()]
  - tid1
  - tid2
  - tid3

I love it!

anandkp's picture

Nice stuff Jesse! The concept is awesome!

I especially like the idea of "Test-Profiles" like what you said here:

The form could have a couple pre-baked combos like WCAG A compliance and AA compliance.

This would make life so much better for folks who are not very conversant with the standards.

It reminds me the W3C/WCAG 2.0 list-tool:

The thought of making the module use the quail.js as it's source of information resonates very much with me! +1 to that concept - I know that it'll put focus on quail.js as the canonical reference and I see that being a big selling point for getting more people from more techno-communities (node.js, ruby, php) involved in supporting it.

Nice stuff!

Let's move this discussion to

jessebeach's picture

Let's move this discussion to Discussion: Removing certain OAC tests, revisiting alignment with WCAG/508 in the QUAIL repo issue queue where it is actionable.

We're making progress on the

jessebeach's picture

We're making progress on the guideline-to-tests mapping!

Great to see! This is a

Hanno's picture

Great to see! This is a greaat overview. Are the orphaned tests really orphaned tests, or still to be mapped?

How to help?

anandkp's picture

Hey there Jesse!

Is there a way for others to help? Is there a discussion we should read to get up to speed?

Have you heard of Maybe we could use it to help organise our async efforts as quail is on Github. :-)

I'm a fan of very light

jessebeach's picture

I'm a fan of very light process. So far, the process has been REALLY light, with the definition of our current development effort held in my head and Kevin's and expressed in issue comments.

This is probably a good time to whip up a simple wireframe of what I've been envisioning so that we can get more folks involved.

anandkp's picture


I was thinking a tool like might help us get more people involved with less code-conflict and better coordination.

I'd love to help with this effort but feel lost on how to contribute. The wireframe you propose sounds great - maybe a hangout to demo it to us?

Who us is also a big question... I'm thinking the Drupal A11y group here as well as on Google+ may be a good start.

I feel awful at having missed the A11y scrum you guys just had... I've put the video of that meet up in my "To Watch" list... ;o)

Lemme know!

A demo is a great idea! Let

jessebeach's picture

A demo is a great idea!

Let me get this wireframe put together. I'll try to do this tonight. Today is planning day at Acquia and I'm splitting my time between two teams this cycle, so I've got a busy day before the evening.


anandkp's picture

Thanks a tonne Jesse!!!

(Spell check Anand... spell check...)