Skip to content

Ensure keys match when comparing reports

Adam Cohen requested to merge ensure-keys-match-when-comparing-reports into master

What does this MR do?

This MR adds a new compare_report_keys() function to the compare_reports.sh script, which ensures that the keys in both the actual and expected reports match. This check is important because fields which contain information that can vary, such as timestamps, are stripped from both reports, so without having a check to ensure the keys are the same, it's possible for one report to be completely missing a field.

This MR also adds a new test-compare-reports-script job which uses rspec to test the logic of the compare_reports.sh script

This has been reverted in !189 (merged). See discussion

What are the relevant issue numbers?

gitlab-org/gitlab#244829 (closed)

Testing

  • New unit tests have been added using rspec

  • Manual test has been performed here to show that if a field is missing, an error is generated in the QA stage:

    $ sh ./compare_reports.sh cs $REPORT qa/expect/$REPORT
    --- qa/expect/gl-container-scanning-report-keys-only.json
    +++ gl-container-scanning-report-keys-only.json
    @@ -61,8 +61,9 @@
           },
           "version": ""
         },
    -    "end_time": "",
         "type": "",
    +    "start_time": "",
    +    "end_time": "",
         "status": ""
       }
     }
    Actual report gl-container-scanning-report.json differs from
    expected report qa/expect/gl-container-scanning-report.json: keys do not match
  • Manual test to ensure existing QA tests still function as expected here

Does this MR meet the acceptance criteria?

Edited by Fabien Catteau

Merge request reports