Update `vulnerability_reads` scanner in the ingestion pipeline
What does this MR do and why?
As described in the linked issue, the scanner data can be inconsistent
between the vulnerability_reads and vulnerability_occurrences
tables for the same vulnerability.
This should never be the case, as they should always be referring to the same data. This is because vulnerability_reads is just a de-normalized read-optimized table containing heavily read data from the vulnerability and vulnerability_finding models
The scanner reference is being properly updated for the finding in the
finding_map class.
This change makes it so that the existing vulnerability_reads is also updated with a new scanner reference
MR acceptance checklist
Please evaluate this MR against the MR acceptance checklist. It helps you analyze changes to reduce risks in quality, performance, reliability, security, and maintainability.
Screenshots or screen recordings
| Before | After |
|---|---|
| before | after |
How to set up and validate locally
prerequisites
- a EE enabled local gdk
- ci runners running locally
- clone the example repo
without the MR patch
-
set the
reports/sast.jsonfile in the example project to this:example test vulnerability
{ "schema": "https://gitlab.com/gitlab-org/security-products/security-report-schemas/-/raw/v14.0.0/dist/sast-report-format.json", "version": "15.0.0", "scan": { "start_time": "2020-08-24T07:30:25", "end_time": "2020-08-24T07:54:47", "status": "success", "type": "sast", "scanner": { "id": "minac", "name": "Minac manual scanner", "url": "https://gitlab.com/minac", "version": "0.0.1", "vendor": { "name": "Minac" } }, "analyzer": { "id": "minac", "name": "Minac manual analyzer", "version": "0.0.1", "vendor": { "name": "Minac" } } }, "vulnerabilities": [ { "message": "Test vulnerability - 0", "category": "sast", "description": "Test vulnerability description", "id": "d1b90d3d-607b-4b92-a68a-552f7ba83deb", "cve": "d1b90d3d-607b-4b92-a68a-552f7ba83deb", "severity": "Critical", "identifiers": [ { "name": "Test CVE identifier", "type": "CVE", "value": "CVE-2023-XXXX" }, { "name": "CWE identifier", "type": "CWE", "value": "CWE-79" } ], "location": { "file": "README.md", "start_line": 0 }, "scanner": { "id": "minac", "name": "Minac" }, "solution": "No solution" } ], "remediations": [] } -
run the pipeline for the main branch
- this will seed a vulnerability that has a scanner
external_idof"minac"
- this will seed a vulnerability that has a scanner
-
run a pipeline again
-
find the vulnerability in the rails console
-
compare
vulnerability.finding.scanner.external_idandvulnerability.vulnerability_read.scanner.external_id# find the vulnerability we seeded vulnerability = Vulnerability.find(541) # compare the scanner references. Without the patch, they will be # different. With the patch, they will match vulnerability.finding.scanner.external_id # => "minac-new-version" vulnerability.vulnerability_read.scanner.external_id # => "minac"
with the MR patch
- follow the same steps as above, though this time you should see the
vulnerability.vulnerability_read.scannerandvulnerability.finding.scannerstay in sync whenever you edit the scanner id inreports/sast.json
Related to #442929 (closed)
Changelog: fixed
EE: true
