Discussion: Support dynamic output of checks / parsing output / more than just an exit code to help customers understand "what's next?" when a check fails

Today, checks either pass or they fail. This is not always the best user experience. There are cases where it's more helpful to give additional details about the failure.

Consider a customer looking at the output of a particular check and wondering "What's next?".

Generic Example

If the check is "identify projects that have x setting turned on", today we tell them "this check failed". We do not say "this check failed because these specific projects have x turned on".

Today, we have to 🤞 hope that the reference or workaround URLs have more information about what customers should do next to identify the specific projects that are leading to the failure. Providing those projects in the output of the check would be helpful and help customers self-serve the results of the check.

A Plan + Questions

On 2025-07-07, Elif and I discussed how we might approach this effort.

conservative default or informative default

On 2025-07-08, Ulises and I discussed whether we should have a conservative default or an informative default.

• conservative default: return the number of items that cause a check not to pass instead of returning project IDs (or similar).
• informative default: return the project IDs

If we chose the conservative default, the user would need to run GitLab Detective a second time to get the project IDs. Or they would need to run a command or script specific to the failing check.

• Should we use a conservative or an informative default?
  • Can we decide this on a per-check basis?
  • Should we decide this on a per-check basis?

Do we change the checks or add to the checks?

In order to accomplish the point of this issue, should we change the command value in the checks that exist today? Or should we add a new field that collects the dynamic output?

💬 Consider that we use the exit code currently to determine whether a message value appears in the report. If we change the command value to complete this issue, we will have to identify another way to control whether a message value appears in the report.

How would each check need to be adjusted?

We have to figure this out regardless of the answer to the above?

• Let's consider each type of check.
  • What adjustments would need to be made to each type of check to make this go?
    • What patterns would we need to learn to adjust the current SQL queries to report project IDs (or similar) (in addition to or instead of 0 or 1)?

🩴 Logic Flip

We should likely refrain from the logic flip discussed in #104 and !118.

• We're likely to "rewrite" the checks as a part of this issue.
• We are used to the current state.

🔭 Vision

Consider this issue (#28) with #124. Together, these items will improve the output that GitLab Detective returns and make it easier to understand.

• Today: "this check did not pass; use the information in the workaround_url and the message to find out more information".
• Tomorrow: "this check did not pass because the projects with these IDs did not meet certain conditions" OR "this check did not pass because of this instance-wide problem".