Move Secure test QA expectations from test projects to analyzers
Problem to solve
As discussed with #33724 (comment 314481785), we currently have an issue with circular dependencies where our devopssecure analyzers require updates and depend on a strict QA comparison to downstream test projects. To quote from the above comment, @fcatteau wrote:
To recap, I see two ways we can ease the pain:
A. change the CI configuration shared by the analyzer projects so that a MR isn't blocked when the generated report doesn't match the expected one, while still making sure the change is properly reviewed
B. move the expected reports to the analyzer projects, so that they are updated and reviewed in the MR where the behavior of the analyzer project is changedA would be a smaller step for the Dependency Scanning projects, where QA using downstream pipelines has already been set up. That said, B is probably easier to set up in the SAST projects, which currently rely on embedded integration tests.
To me, ideally the expected reports should be with the analyzer projects (proposal B), so that they can be reviewed as the analyzer project changes. We would keep external test projects, ensuring we have realistic integration tests. If nothing else, I believe external test projects and downstream pipelines are the only way to test Auto DevOps integration. That said, this doesn't help with running QA jobs for MRs created by the community. See #118871 (closed) (closed) cc @ifrenkel
Intended users
User experience goal
Less developer pain in keeping our test projects up to date.
Proposal
-
Allow analyzers to store expectations for test projects within the analyzer project itself gitlab-org/security-products/ci-templates!84 (merged) -
Update each SAST analyzer with its downstream expectations -
Update each DS analyzer with its downstream expectations -
Remove update QA jobs from QA templates, and from the test projects
Implementation Plan
For each analyzer project:
A. Move test report expectation into analyzer project
B. Specify report to downstream job via **_REPORT_URL within analyzer .gitlab-ci.yml (example)
C. Set **_REPORT_URL within test project .gitlab-ci.yml so default branch can continue to pass (for scheduled pipelines)
See example: gitlab-org/security-products/analyzers/secrets!59 (diffs)
SAST Analyzers
DS Analyzers
Test Projects
Documentation
-
Document testing architecture in handbook, see related gitlab-com/www-gitlab-com!53874 (merged) (gitlab-com/www-gitlab-com!68391 (merged)) -
Update README of tests/common(gitlab-org/security-products/tests/common!24 (merged))