Migrate compare_reports Shell script to ruby

Problem to solve

As discussed in Ensure keys match when comparing reports, it was noted that the compare_reports.sh script is becoming increasingly more complex and difficult to follow.

Now that ruby unit tests are being added as part of Ensure keys match when comparing reports, we should consider converting the compare_reports.sh script to use ruby instead.

The purpose of this issue is to migrate the existing compare_reports.sh to use ruby.

Intended users

• Sasha (Software Developer)

User experience goal

The compare_reports.sh script will be easier to maintain and follow by other developers, and we can add more test coverage using standard ruby tools such as rspec

Proposal

1. Migrate the compare_reports.sh to ruby
2. Update all of the QA YAML files to use the new ruby script/command.
3. Ensure all the existing QA tests still function as expected

The ruby script might still use jq, in which case replacing jq with plain ruby code will be done in #299503 (closed).

Implementation plan

• Add QA script for DS, SD, and SAST to the integration-test project, with tests for these. gitlab-org/security-products/analyzers/integration-test!2 (merged).
• Update the Dependency Scanning QA template to execute these scripts and the integration-test Docker image. gitlab-org/security-products/ci-templates!236 (merged)
• Update README of integration-test project and cover job integration tests gitlab-org/security-products/analyzers/integration-test!5 (merged)

Enabling these scripts for SAST and Secret Detection is out of scope.

• https://gitlab.com/gitlab-org/security-products/ci-templates/-/blob/master/includes-dev/qa-sast.yml
• https://gitlab.com/gitlab-org/security-products/ci-templates/-/blob/master/includes-dev/qa-secret_detection.yml

Further details

compare_reports.sh currently goes through the following steps:

1. sanitize_and_diff_reports
   1. filter the actual and expected reports using jq filters; the exact filter depends on the report type 
   2. compare the actual report to the expected report using diff -u -b
   3. report an error and exit 1 if there's a diff
2. validate_report_json
   1. get the commit ref of security-report-schemas based on the version stored in the actual report
   2. get the filename of the schema based on the type of the actual report
   3. fetch the JSON schema
   4. do JSON schema validation using Python
   5. report an error and exit 65 if validation fails

These steps have already been ported to RSpec shared examples in gemnasium!194:

• it_behaves_like recorded report compares the generated report with the expected one
• it_behaves_like valid report performs schema validation

But there are limitations:

• These are RSpec shared examples, not a Ruby script that could be used in place of compare_reports.sh.
• Report sanitization is specific to Dependency Scanning reports. See #299503 (closed)
• Schema validation uses a local copy of the schema whereas compare_reports.sh fetches the schema that matches the report type and version.

Availability & Testing

Unit tests will be added for the new ruby script.

What does success look like, and how can we measure that?

The compare_reports.sh script is replaced by a ruby script with unit tests.

What is the type of buyer?

GitLab Ultimate Enterprise Edition

Is this a cross-stage feature?

Yes, this affects all secure projects

Links / references

Add tests for compare_reports script in ci-templates project

/cc @NicoleSchwartz @gonzoyumo @fcatteau @theoretick