Continuous accessibility testing

Motivation

In the light of &31 I thought it might be smart to put continuous accessibility testing on the radar by laying out some of the options we have and to set up a small PoC.

I know that there are some existing issues on selenium, but those are mostly about continuous cross-browser testing:

Continuous accessibility testing is exactly as it implies, it runs a test like any other (think linting) but for accessibility definitions/practices.

Problem/Value

Even with our efforts around implementing the GitLab design system things might fall through the cracks and/or this might help catch/solve things sooner. Accessibility isn't a checklist, but these tests can automate catching/solving the simpler ones, so we humans can focus on the rest.

Options

  • Selenium - Open source - Automates browsers, so not directly accessibility related
  • Accesslint - Closed source, only works on GH
  • Tenon.io - Closed source
  • Achecker - Online web accesibility tool - not relavant for CI
  • Pa11y - Open source - Has native api support for rendering dashboards and json outputs
  • Axe core - Open source - Relies on code injection in webpages - might be usable with selenium
  • Google Chrome accessibility developer tools - Open source - can be used with selenium
  • Paypall AATT - Open source - yet to look into this one, need dev assistance
  • Accessibility.js - Open source - Relies on code injection in webpages - might be usable with selenium
  • Tanaguru - Open source - Online environment to test pages, prob not CI approachable
  • The A11y Machine - Open source - Outputs a dashboard website displaying those errors, possible to post towards gitlab pages.
  • Tota11y - Open source - Relies on code injection in webpages - might be usable with selenium

Proof of Concept

Pa11y seems the best/most attainable to implement in order to set up a PoC (makes great use of our review apps).

It has a dedicated CI npm package and lot's of configurability (should even support screenshots, although that may only work for the non-CI CLI tool, perhaps possible to use that instead of dedicated CI tool). It supports sitemaps and has options to fail the tests above a certain error rate.

As the .gitlab-ci.yml file of gitlab-ce is far too complicated and Pa11y needs a live/deployed web app/site in order to test against, I went for testing about.gitlab.com.

I created the following MR gitlab-com/www-gitlab-com!10094 (closed)

The test currently should output all kinds of errors (notices, warning, critical errors), but it should be possible to set up a highly customisable testing process.

It outputs a json output, which I piped into a file (see this job's artefacts)

An example out the output may look like:

      {
        "code": "WCAG2AA.Principle4.Guideline4_1.4_1_1.F77",
        "context": "<input class=\"st-default-search-input st-search-set-focus\" type=\"text\" value=\"\" placeholder=\"Search this site\" id=\"st-overlay-search-input\" autocomplete=\"off\" autocorrect=\"off\" autocapitalize=\"off\">",
        "message": "Duplicate id attribute value \"st-overlay-search-input\" found on the web page.",
        "selector": "#st-overlay-search-input",
        "type": "error",
        "typeCode": 1
      },

References

Edited by Dimitrie Hoekstra