Leverage scan object to provide QA test scan duration check for Analyzers
Problem to solve
We don't track the duration of the analysis for the following projects and it could drastically increase it without being aware of it:
#34903 (closed) solved this issue by adding the duration check to the integration test stage of each project, however, the above projects were not updated because they don't have an integration test stage.
Once Add scan object to the Security Reports JSON format has been completed (at least, when we have the necessary data in the JSON report), we can leverage this scan object to provide scan duration check for the above projects.
Intended users
devopssecure team members
Proposal
To make sure we don't increase analysis duration unintentionally after an update, we need to track it and ensure they stay within a given threshold.
When Add scan object to the Security Reports JSON format has been completed, we need to do the following:
-
Introduce a MAX_SCAN_DURATION_SECONDS
variable to the following.gitlab-ci.yml
files: -
Add logic to the following qa template files to determine the scan duration time by using the scan.start_time
andscan.end_time
values provided by the scanning report, and ensuring that this value is less than theMAX_SCAN_DURATION_SECONDS
value:-
Dependency Scanning -
SAST (blocked by #33724 (closed)) -
Secret Detection
-
-
Remove the check analyzer scan duration
andintegration test
jobs inincludes-dev/analyzer.yml
. Please validate with groupstatic analysis first as they currently still use the integration tests for community contributions MR (which can't run downstream QA pipelines): removal MR. -
Test the following projects to ensure that a scan which exceeds the MAX_SCAN_DURATION_SECONDS
causes a failure in the downstream qa test:
Documentation
-
This should be added as part of our test projects documentation: https://gitlab.com/gitlab-org/security-products/tests/common#security-products-test-projects MR
Testing
Try to make the QA fail by running an analysis that gets over the threshold.
SET to look at creating branches, add a before_script
with a sleep
command. This should fail as it has exceeded the specified maximum duration.
What does success look like, and how can we measure that?
QA Pipeline fails when an analysis duration exceeds the allowed variation.
Note
Duration scan checks not added to DAST and License Scanning as they use a different structure for testing. Container Scanning changed to an integration test structure as well (more info).