Create linter tests to validate updates to Vale/Markdownlint
Issue
When an update to Vale or Markdownlint starts catching new violations, it's easy to notice because the pipeline fails when we try to update it with the new image (in gitlab
and other projects).
But when a linter breaks and stops working, it's extremely hard to notice the problem. This happened just recently where a newer version of Vale stopped catching one violation, but we didn't notice because the pipeline remained green. I only noticed because I forgot to update my local version, and a doc I opened just happened to have an example that was failing the vale rule locally (with the old version), but passing in master
with the new Vale version.
See related upstream issue: https://github.com/errata-ai/vale/issues/657
Related to Fix curl examples missing quotes (gitlab-org/gitlab!125495 - merged)
Proposal
We should create some fixtures (test docs) and populate them with markdown that is expected to fail the linting. The script that runs the tests can count the failures, and if the number does not match the expected value, we'll know that the new version is failing in some way.