Display axe-core's "incomplete" violations report in components' accessibility tests
The problem
While reviewing an MR, I decided to verify how effective is axe at detecting color-contrast violations in component’s states like focus and hover. As you know, we encourage testing these states. I found that some color-contrast violations are reported as “incomplete” by axe. “Incomplete” or “Needs review” violations are those that axe can’t fully confirm (they might be false positives). The `cypress-axe` package (that we use to run axe checks in tests) ignores “incomplete” violation reports therefore we can’t access this data to implement custom assertions. You can verify this infomation in cypress-axe
source code: https://github.com/component-driven/cypress-axe/blob/master/src/index.ts#L90
Solution proposal
If we want to have access to "incomplete violations" reports, we need to replace cypress-axe
with an alternative that allows us to access this information. My high-level assessment is that we could implement an alternative glCheckA11Y
command that interacts directly with the axe-core library instead of using cypress-axe
as a mediator.
Running axe-core
directly will expose "incomplete" reports and we can run assertions on these results.