Surface docs linting warnings in MR diffs with code quality
Proposal
Currenly, vale errors, warnings, and suggestions are hidden from users that don't use vale locally. Vale is an incredibly important tool for the technical writing team, and the feedback we get from the tool helps improve the quality of our docs.
Based on https://docs.gitlab.com/ee/user/project/merge_requests/code_quality.html#implementing-a-custom-tool, I realized we can output vale JSON that code quality supports, which will start surfacing these warnings in MRs.
I expect this to be a huge improvement for public contributors, because the vast majority do not have vale installed locally. Many work entirely from GitLab and have no access to these messages.
It's also a great tool for internal contributors as well, because any vale error that fails the pipeline will be highlighted in the diff, along with a message explaining how to fix the failure. This should speed everyone up as they no longer need to search the logs for the error, then search the files for the correct line.
Screenshot
POC MRs
-
Sample MR with only docs changes, you can see the changes shown in the screenshot above (scroll down to the
index2.md
test file in the diff). Hover over each icon to see the vale message: !101773 (diffs) -
Sample MR with BOTH "code" code quality warnings and "docs" code quality warnings, working together. Scroll down to the
index2.md
file to see the docs warnings, and also to L143 inlib/gitlab/database/migrations/constraints_helpers.rb
to see the code warnings: !101857 (diffs). These messages were generated from two different jobs, but they can work together thanks to thegraphql_code_quality_full_report
feature flag, enabled in thegitlab
project, see: #340525 (closed)Also, check the MR widget in the overview, as it shows all the messages in the code quality widget. Currently shows as all degradations because there is no code quality artifacts for docs in
master
to compare against:
MVC Steps:
- Code quality relies on comparisons to artifacts in the latest pipeline on
master
. We need to make sure that the majority of pipelines onmaster
run code quality, so there is always an artifact available to compare against: !101856 (merged) (this also fixes just a general bug with code quality in thegitlab
repo, which is a nice side effect). - Swap the double quotes in the vale messages for single quotes, otherwise the JSON will be incorrectly formatted: !101887 (merged) (Also sets the warning levels in the diff)
- Add the
code_quality_docs
job itself. Make sure:- It is allowed to fail, and always returns the code quality artifact.
- Runs in all MR pipelines that contain docs, and all pipelines on master so we can always compare against
master
. - At this point, roll it out for
error
andwarning
level issues first, and only ingitlab
(not contributor's forks).
- Expand the job to
suggestion
level issues, and possibly to fork pipelines for contributors. - Future brainstorming 1 - Use the bots to add/expand the messaging we already throw into docs MRs that explains how to make use of the diff warnings.
- Future brainstorming 2 - Output the links to the documentation pages that relate to the rules. Alternative, link to a single page that offers general advice to contributors?