Iteration 1: Implement metrics to measure the number of broken Codeowner files and the failure modes on SaaS based on basic linter

Why do this?

We need to better understand to which extent Codeowner files are potentially broken. If it is significant we shall attempt to reduce this with additional work proposed in &4498 (closed). We will continue to track this metric and see if this work pays of in results.

Proposal

• Use the basic Codeowners linter to go through all projects in and check every time an MR is made on GitLab.com that requires codeowner approval if the codeowner file is broken or not and thereby assess
  • the total number of actively used broken Codeowner files vs. the total number of projects with actively used Codeowners
  • detail the statistics with number of actively used broken Codeowner files per failure mode:
    • missing owners in entry (docs)
    • sections without a name (docs)
    • required approvals for optional sections (docs)
    • invalid section format (docs)
• Create a report in Sisense that collects these metrics that report on the occurrence of different failure modes as well as the total number of actively used broken files
  • Focus on SaaS only. We can assume that any improvement on SaaS will happen similarly on SM.
  • Include counting the total number of actively used codeowner files and track the change over time.

Dependencies

• Add a validator Ruby class to validate CODEOWNERS (#396476 - closed) • Vasilii Iakliushin • 15.11 • At risk
• Document CODEOWNER syntax and logic (&10057 - closed) • Unassigned

Implementation proposal (by @vyaklushin)

Gitlab::CodeOwners::File defines valid? and errors methods to verify if the file has a correct syntax.

We can try to inject this verification into MergeRequestApprovalPresenter to run the check on each merge request view.

For each error we track a snowplow event with project id and error type.