Remove unnecessary :SKIP: from all expectations
What does this MR do?
When image integration tests were added to this project, the following variable fields in the qa/expect
files were marked with the :SKIP:
directive:
vulnerabilities[].id
scan.analyzer.version
scan.scanner.version
scan.start_time
scan.end_time
However, adding :SKIP:
to these fields is unnecessary, since they're automatically ignored by the GitlabSecure::IntegrationTest::Comparable module:
-
vulnerabilities[].id
is removed here -
scan.analyzer.version
is removed here -
scan.scanner.version
is removed here -
scan.start_time
is removed here -
scan.end_time
is removed here
Unnecessarily adding these :SKIP:
directives leads to the following:
-
Confusion, since they're not necessary and serve no purpose.
-
Extra maintenance cost, both from developers needing to remember to add the
:SKIP:
directives, as well as reviewers needing to ensure that MRs contain these:SKIP:
directives. -
High likelihood for expectation files to become inconsistent, with some entries containing
:SKIP:
directives, and others that do not, because the process is manual and error-prone. We should instead use an automated process to handle this, which is exactly what theintegration-test
project does by programmatically ignoring specific fields. -
It's not consistent with other analyzers in this stage. For example:
-
mobsf does not use
:SKIP:
in any expectations. -
gemnasium only uses
:SKIP:
where it's absolutely necessary, as intended. -
security-code-scan does not use
:SKIP:
in any expectations. -
sobelow does not use
:SKIP:
in any expectations. -
flawfinder
uses:SKIP
in some expectations but not all. -
pmd-apex does not use
:SKIP:
in any expectations. -
brakeman does not use
:SKIP:
in any expectations.
This inconsistency makes it difficult for developers to switch between analyzers in different groups, because they need to understand which analyzers expect the
:SKIP:
keyword to be added to all variable fields in all expectations, and which ones do not. This slows down code reviews by needlessly discussing and defending decisions which should've be left to automated processes. -
mobsf does not use
-
It makes it more difficult to automate the process of refreshing expectations, since we need to ensure the new expectation files copy the
:SKIP:
directives in the appropriate locations.
This MR removes all of these unnecessary :SKIP:
directives.
What are the relevant issue numbers?
N/A
Does this MR meet the acceptance criteria?
- [-] Changelog entry added
-
Documentation created/updated for GitLab EE, if necessary -
Documentation created/updated for this project, if necessary -
Documentation reviewed by technical writer or follow-up review issue created -
Tests updated/added for this feature/bug -
Job definition updated, if necessary -
Conforms to the code review guidelines -
Conforms to the Go guidelines -
Security reports checked/validated by reviewer