Configure default a11y scanning template to scan changed pages

Problem to solve

Right now the Accessibility CI template accepts a configuration-defined variable of URLs to pass into the pa11y scan. To make it easier to automatically scan relevant pages, we should be able to detect which ones have changed and only scan those.

Intended users

Proposal

We considered making changed pages and URLs (via routemap.yml) available as CI variable, but decided against it because of performance issues.

Instead, we opted to use git diff --name-only in the CI environment and work with the diff there, because it's much cheaper to execute, and in turn much safer while also being in a contained environment. This feature might use a similar function. If there's a few logic pieces involved (e.g. detect+parse routemap, generate diff list, match, filter for visual pages) we might be able to package it nicely as a command-line tool available in the accessibility docker image.

I propose we add a second job to the Accessibility template that's the same as the current one, but uses CI-based diffing logic and the following rules

rules:
  - if: $CI_MERGE_REQUEST_ID

We can also add

rules:
  - if: $CI_MERGE_REQUEST_ID
    when: never
  - when: on_success

to the existing template so when included, an a11y job will still return but with the added step of calculating the changes pages before invoking pa11y-ci.

If a a review app deploys and no changes match the routemap, and in turn no URLs are passed into the pa11y script, a clean report will be produced with 0 violations:

$ pa11y-ci --json
{"total":0,"passes":0,"errors":0,"results":{}}

Original, scrapped proposal

I propose we add a second job to the Accessibility template that's the same as the current one, but uses $CI_MERGED_REQUEST_CHANGED_PAGE_URLS. We provide a template for enabling accessibility scans, and once the changed pages are available in a MR CI variable !25256 (merged) we can use the page urls to automatically scan just the pages that have changed in the merge request. This reduces the amount of noise in the A11y report and gives the author a clearer view of the violations and warning pertaining to their particular change.

Permissions and Security

I'm not currently aware of any security concerns around this work.

Documentation

• Update the docs for a11y to include how the new template and variable work
• Update the docs about how to extend the template to scan not just changed pages/paths
• Update the docs for how to override completely the variable and scan only specified pages.

Availability & Testing

Generally, and end-to-end test would be nice for this feature, so we know that report artifacts are consistently and correctly produced. Testing might be easier after we create our own A11y docker image project in the codequality group, removing most of the script from this template so we can really just test that the MR logic is sound. @zeffmorgan Do you have any thoughts on putting tests on this?

What does success look like, and how can we measure that?

When someone includes Verify/Accessibility.gitlab-ci.yml in their configuration file, all of their pipelines, branch and merge request, produce useful a11y reports that don't require additional configuration. We can measure this by seeing how many people include the template on their project without overriding any of the template job configuration. Is it possible to track that?

What is the type of buyer?

Links / references