Automated a11y scanning of Review Apps MVC
Problem to solve
Performing accessibility testing is important in order to ensure you're serving all the users who use your products. We offer code quality validation automatically via codeclimate by scanning code at build time, we could also provide a comprehensive accessibility scan by running https://github.com/liip/TheA11yMachine or https://github.com/pa11y/pa11y-ci against a built web application as deployed to a review app.
NOTE: Throughout this issue accessibility will be shortened to a11y.
Intended users
- Sasha (Software Developer)
- QE/SDET users
Further details
For Sasha:
- When Sasha's is merging code, they want to know if that code triggers an a11y issue, so that they can fix it without manual ally tests.
For QE:
- When a new build is available, they want to know if it has regressed/improved in a11y, so that they can streamline manual a11y tests.
Proposal
Add definition of an accessibility job following the same rough pattern that has been established for code quality and web performance. This gives us the benefit of using existing art and later enabling users to customize the job as needed. A template is NOT required for this MVC, but could be an easy next step after this. We will create a template for inclusion of this job as part of the MVC. A user will need to specify the URl(s) being scanned for accessibility in this iteration.
If a user has added the accessibility job, upon successful deployment of a review app, use one of the a11y scanning tools and the route map (if necessary) to scan the review app for accessibility issues, which can be reported as a CI artifact for the MVC.
The first implementation of this will use pa11y-ci and produce a job artifact that can be viewed in its entirety, or diffed against another artifact of the same format in the context of a merge request to see newly introduced violations. pa11y-ci is very simple to implement first, and is flexible enough to use axe or be adapted to use a11ym.
The first version of the job artifact will be a superset of the pa11y-ci report, containing all of its data but also allowing for additional sections with metadata or other report formats. This includes all errors, warnings and notices as documented in the pa11y docs.
In the future we will allow for better ways to pass variables to the job. A fast follow-on to this issue will be to dynamically capture changed path paths and changed page urls so the job does not need to be changed with each build.
Permissions and Security
Documentation
This is a new feature and first entry into a new category. At a minimum the new documentation should:
- Show a user how to enable this feature in their gitlab-ci.yml
- Show a user what to expect on their first and subsequent MRs after enabling the feature.
- Direct a user to documentation to help them resolve issues from the report (if not provided in the report).
Testing
What does success look like, and how can we measure that?
Acceptance Criteria
- Full a11y report is available to user as an artifact in the new job
Success Criteria
- Have merge requests use this from the start instead of implementing a separate tooling for a11y testing.
- Identify accessibility improvements that were discovered as part of this new auto scanning step in a downloadable report.
Out-of-scope
- Merge request widgets and report-rendering UI will be addressed in separate issues
What is the type of buyer?
The buyer for this feature is the individual contributor who wants to do some level of a11y testing. As such this will be a Core feature.
Links / references
Items to be addressed later:
This page may contain information related to upcoming products, features and functionality. It is important to note that the information presented is for informational purposes only, so please do not rely on the information for purchasing or planning purposes. Just like with all projects, the items mentioned on the page are subject to change or delay, and the development, release, and timing of any products, features, or functionality remain at the sole discretion of GitLab Inc.