Update compare_reports.sh script to return an error if JSON validation fails

Problem to solve

During development of Ensure QA report keys match for secure analyzers, it was decided in this discussion that instead of the QA stage failing if the JSON validation fails, we should instead print out a warning message. This will allow a "phased rollout" so other teams have time to address the issues. After a certain amount of time, possibly 2 weeks, we'll change the behaviour of the compare_reports.sh script so that an error will be returned if the JSON validation fails. The purpose of this issue is to make that change.

Intended users

• Sasha (Software Developer)

User experience goal

GitLab secure team will be able to quickly identify if a generated report is producing JSON which fails validation.

Proposal

1. Inform the secure team that we'll be failing the compare_reports.sh script if JSON validation fails
2. Remove the allow_failure block from the following QA templates:
   1. includes-dev/qa-container_scanning.yml See gitlab-org/security-products/ci-templates!224 (merged)
   2. includes-dev/qa-dependency_scanning.yml
   3. includes-dev/qa-sast.yml
   4. includes-dev/qa-secret_detection.yml
3. Test each of the above templates to ensure that they fail when the JSON report can't be validated against the security reports schema; see discussion

Further details

These changes will help reduce production bugs related to the format of the generated security reports

Availability & Testing

Testing has already been implemented as part of Ensure QA report keys match for secure analyzers

What does success look like, and how can we measure that?

The QA stage will fail if the analyzer report cannot be validated against the security report schema

What is the type of buyer?

GitLab Ultimate Enterprise Edition

Is this a cross-stage feature?

Yes, this affects all secure stage products

Links / references

#244829 (closed)

/cc @gonzoyumo @NicoleSchwartz @fcatteau