Follow-up for newly introduced validity checks

Please note that gitlab-org#23343 supersedes some of the items in this issue. Closed in favour of the epic.

Overview

This tracks a list of follow-ups for newly introduced validity checks, carried over from this status update (not in a particular order):

  • Restrict new validity checks to GSS analyzer findings only (issue - draft MR).
  • Ensure token type extraction works if GSS findings drop gitleaks identifers - to address !249626 (comment 3714532458).
  • Remove the pattern check from all validity checks – to address !248110 (comment 3658223808).
  • Move status code mapping into the base client – to address !248110 (comment 3658223755).
  • Address all deviations between the generator we build during the challenge and the existing validity checks.
  • Ensure re-generations produce the same code we have, with an empty diff against master as acceptance criteria.
  • Improve existing instrumentation to know when a check did not call a vendor.
  • Improve existing instrumentation to count how many secrets were not validated because of no vendor support.

And depending on the outcome of secret-detection-rules!219 – that aims to fix !248113 (comment 3681547617):

If new rules are introduced for Slack to replace existing rule:

  • Confirm the new rules store whole tokens in findings and not only the workspace IDs.
  • Update !248113 (closed) to add validity checks for the new rules instead.
  • Add user-facing messaging that findings detected before the fix must be re-scanned to become verifiable1.

If the existing rule Slack has its pattern updated:

  • Confirm the new rules store whole tokens in findings and not only the workspace IDs.
  • Update !248113 (closed) to match the pattern(s) of the updated rule.
  • Add user-facing messaging that findings detected before the fix must be re-scanned to become verifiable2.

  1. In case the existing Slack rule is replaced, existing findings would still carry that rule ID and would never trigger a validity check. Might be okay if we never introduce a validity check for that particular rule though. ↩

  2. In case the existing Slack rule has its pattern updated, existing finding will keep the truncated value until re-scanned, so we need to let the users know, and offer them a workaround, e.g. re-scanning so the value is updated. ↩

Edited by Ahmed Hemdan