Technical Discovery: E2E testing for Static Analysis analysers
Context
Analysers maintained by groupstatic analysis undergo two kinds of testing:
- Integration tests which exercise the analyser against a project fixture, designed to represent a repository being scanned in CI. This used to involve downstream tests (actually running in a CI pipeline), but has now been converted to use the integration-test runner as part of #366852 (closed). These tests perform a scan, and then compare the resulting
gl-*-report.json
against the corresponding expected report. - Unit tests, which exercise select parts of the Go wrapper code, such as report processing, rule filtering, ID mapping etc.
We had a tiny bit of E2E test coverage previously due to the way downstreams were run in CI. However this is no longer the case, and there's now a gap we should close.
Proposal
This discovery issue aims to address the following:
- Decommissioning the secure-test-project-orchestrator for all analysers owned by groupstatic analysis, because it's just noise at this point. There are daily QA failures because the downstream repos these tests run against haven't been updated after the migration to
integration-test
. The tests don't provide any value on top of the integration tests maintained alongside each analyser.
- Creating new E2E tests which exercise analysers based on specific customer journeys. Taking Category:Secret Detection for example, a journey could involve creating a project, enabling historical scans, running SD, and observing the results. For a SAST analyser, custom rulesets and #368284 (closed) could be tested in tandem. These tests might run in the production environment to serve as an extra layer of validation for a merged MR, and can also be designed to run locally via the GDK. Tools like Playwright can be used to define journeys and execute these tests.
🤖
Auto-Summary Discoto Usage
Points
Discussion points are declared by headings, list items, and single lines that start with the text (case-insensitive)
point:
. For example, the following are all valid points:
#### POINT: This is a point
* point: This is a point
+ Point: This is a point
- pOINT: This is a point
point: This is a **point**
Note that any markdown used in the point text will also be propagated into the topic summaries.
Topics
Topics can be stand-alone and contained within an issuable (epic, issue, MR), or can be inline.
Inline topics are defined by creating a new thread (discussion) where the first line of the first comment is a heading that starts with (case-insensitive)
topic:
. For example, the following are all valid topics:
# Topic: Inline discussion topic 1
## TOPIC: **{+A Green, bolded topic+}**
### tOpIc: Another topic
Quick Actions
Action Description /discuss sub-topic TITLE
Create an issue for a sub-topic. Does not work in epics /discuss link ISSUABLE-LINK
Link an issuable as a child of this discussion
Last updated by this job
- TOPIC Decommissioning the test orchestrator for Static Analysis analysers #408944 (comment 1404761908)
Discoto Settings
---
summary:
max_items: -1
sort_by: created
sort_direction: ascending
See the settings schema for details.