Add scan duration check for Analyzer integration tests
Problem to solve
We don't track the duration of the analysis and could drastically increase it without being aware of it.
Intended users
devopssecure team members
Proposal
To make sure we don't increase analysis duration unintentionally after an update, we need to track it and ensure they stay within a given threshold.
-
add an expected duration in seconds, defined by a MAX_SCAN_DURATION_SECONDS
environment variable -
add a comparison logic in the QA jobs -
make the QA fail when duration increase past the expected duration number of seconds. -
apply to the existing tests projects
Documentation
This should be added as part of our test projects documentation: https://gitlab.com/gitlab-org/security-products/tests/common#security-products-test-projects
Testing
Try to make the QA fail by running an analysis that gets over the threshold.
What does success look like, and how can we measure that?
QA Pipeline fails when an analysis duration exceeds the allowed variation.
Links / references
Projects required to be updated
-
Kubesec -
Retire.js -
Sobelow -
Spotbugs - merged despite the fact that it's blocked by #39120 (closed)
-
Secrets -
Tslint -
eslint -
gemnasium- blocked by gitlab-org/security-products/ci-templates!57 (merged)
- see #196697 (closed) for workaround
-
gemnasium-maven- blocked by gitlab-org/security-products/ci-templates!57 (merged)
- see #196697 (closed) for workaround
-
gemnasium-python- blocked by gitlab-org/security-products/ci-templates!57 (merged)
- see #196697 (closed) for workaround
-
bundler-audit- blocked by gitlab-org/security-products/ci-templates!57 (merged)
- see #196697 (closed) for workaround
-
gosec -
nodejs-scan -
security-code-scan -
phpcs-security-audit -
brakeman -
flaw finder -
klar -
dast -
pmd-apex -
find-sec-bugs
Edited by Adam Cohen