Skip to content

Remove unnecessary :SKIP: from all expectations

Adam Cohen requested to merge remove-skip-directive-from-all-expectations into main

What does this MR do?

When image integration tests were added to this project, the following variable fields in the qa/expect files were marked with the :SKIP: directive:

  • vulnerabilities[].id
  • scan.analyzer.version
  • scan.scanner.version
  • scan.start_time
  • scan.end_time

However, adding :SKIP: to these fields is unnecessary, since they're automatically ignored by the GitlabSecure::IntegrationTest::Comparable module:

  • vulnerabilities[].id is removed here
  • scan.analyzer.version is removed here
  • scan.scanner.version is removed here
  • scan.start_time is removed here
  • scan.end_time is removed here

Unnecessarily adding these :SKIP: directives leads to the following:

  1. Confusion, since they're not necessary and serve no purpose.

  2. Extra maintenance cost, both from developers needing to remember to add the :SKIP: directives, as well as reviewers needing to ensure that MRs contain these :SKIP: directives.

  3. High likelihood for expectation files to become inconsistent, with some entries containing :SKIP: directives, and others that do not, because the process is manual and error-prone. We should instead use an automated process to handle this, which is exactly what the integration-test project does by programmatically ignoring specific fields.

  4. It's not consistent with other analyzers in this stage. For example:

    This inconsistency makes it difficult for developers to switch between analyzers in different groups, because they need to understand which analyzers expect the :SKIP: keyword to be added to all variable fields in all expectations, and which ones do not. This slows down code reviews by needlessly discussing and defending decisions which should've be left to automated processes.

  5. It makes it more difficult to automate the process of refreshing expectations, since we need to ensure the new expectation files copy the :SKIP: directives in the appropriate locations.

This MR removes all of these unnecessary :SKIP: directives.

What are the relevant issue numbers?

N/A

Does this MR meet the acceptance criteria?

Edited by Adam Cohen

Merge request reports