Use `finding_data` to generate vulnerabilities response
What does this MR do and why?
This MR introduces a new finder class to utilize the recently introduced column called finding_data
instead of parsing the report artifacts for each HTTP request. We will still be parsing the report artifact to access the remediations but only the relevant parts of the artifact will be downloaded and parsed if the finding has remediations.
Summary of the changes
As this MR is a bit bigger than usual, I wanted to summarize the changes to make it easier to review. There are 3 types of changes;
Security::PipelineVulnerabilitiesFinder
1) Stop using the old This old finder class does not obey the current retention policy and keeps serving the finding
records by parsing the CI job artifacts for each request after the retention period. This change could be done in a separate MR on its own but the remaining part will still be huge.
Security::PureFindingsFinder
2) Introduce a new finder called This finder class inherits from the Security::FindingsFinder
to support the same filtering/sorting options. The only difference between the new finder and the parent finder is that the new finder returns Security::Finding
s collection which has all the data available to populate the HTTP response.
3) Switch between the old and new finders
This MR introduces a new feature flag as a safety relay to switch between the new and old finder classes.
4) Read artifact fragments to access the remediations data
As we are storing the byte offsets of each remediation object in the CI job artifact, we can now read the job artifact partially and parse those fragments to access the remediation data.
Related to Use the `finding_data` in the `Security::Findin... (#376215 - closed).
Database review
The new finder inherits from an existing one and uses exactly the same queries. Here is the list of queries;
Load `security_findings`
SELECT
"security_findings".*
FROM
"security_findings"
INNER JOIN "security_scans" ON "security_findings"."scan_id" = "security_scans"."id"
INNER JOIN "security_scans" "scans_security_findings" ON "scans_security_findings"."id" = "security_findings"."scan_id"
WHERE
"security_scans"."pipeline_id" = 129
AND "security_findings"."deduplicated" = TRUE
AND "security_scans"."latest" = TRUE
AND "security_scans"."status" = 1
AND "security_findings"."confidence" IN (1, 2, 3, 4, 5, 6, 7)
AND "security_scans"."scan_type" IN (1, 2, 3, 4, 5, 6, 7, 8)
AND "security_findings"."severity" IN (1, 2, 4, 5, 6, 7)
ORDER BY
"security_findings"."severity" DESC,
"security_findings"."id" ASC
LIMIT 20 OFFSET 0
Preload `security_scans`
SELECT
"security_scans".*
FROM
"security_scans"
WHERE
"security_scans"."id" IN (56, 55)
Preload `ci_builds`
SELECT
"ci_builds".*
FROM
"ci_builds"
WHERE
"ci_builds"."type" = 'Ci::Build'
AND "ci_builds"."id" IN (1168, 1169)
Preload `ci_job_artifacts`
SELECT
"ci_job_artifacts"."project_id",
"ci_job_artifacts"."file_type",
"ci_job_artifacts"."size",
"ci_job_artifacts"."created_at",
"ci_job_artifacts"."updated_at",
"ci_job_artifacts"."expire_at",
"ci_job_artifacts"."file",
"ci_job_artifacts"."file_store",
"ci_job_artifacts"."file_sha256",
"ci_job_artifacts"."file_format",
"ci_job_artifacts"."file_location",
"ci_job_artifacts"."id",
"ci_job_artifacts"."job_id",
"ci_job_artifacts"."locked",
"ci_job_artifacts"."partition_id"
FROM
"ci_job_artifacts"
WHERE
"ci_job_artifacts"."job_id" IN (1168, 1169)
Preload `vulnerability_scanners`
SELECT
"vulnerability_scanners".*
FROM
"vulnerability_scanners"
WHERE
"vulnerability_scanners"."id" IN (15, 12)
Preload `vulnerability_feedback`
SELECT
"vulnerability_feedback".*
FROM
"vulnerability_feedback"
WHERE
"vulnerability_feedback"."finding_uuid" IN ('d4e4c243-6d59-5e75-8f95-0a0ab61b0da2', 'ee6f3d6c-481e-5002-adfd-364594e7cad2', 'f80a8203-a5ac-5699-a87e-609071041c70', 'eb78798f-058d-5665-9ccc-e4a986598a24', '2eb3ccdf-6217-56d0-a047-baab045f8a8a', '2e87cc0d-5d39-5d12-8ecc-be4dba1c38c3', 'c105de26-355f-5982-8e54-e104e95c648f', '8aeb44e7-a63d-5ec5-88ad-7703ff6b0cc4', '29b4a375-280e-5939-8577-faff77c5129c')
Preload `vulnerability_reads`
SELECT
"vulnerability_reads".*
FROM
"vulnerability_reads"
WHERE
"vulnerability_reads"."uuid" IN ('d4e4c243-6d59-5e75-8f95-0a0ab61b0da2', 'ee6f3d6c-481e-5002-adfd-364594e7cad2', 'f80a8203-a5ac-5699-a87e-609071041c70', 'eb78798f-058d-5665-9ccc-e4a986598a24', '2eb3ccdf-6217-56d0-a047-baab045f8a8a', '2e87cc0d-5d39-5d12-8ecc-be4dba1c38c3', 'c105de26-355f-5982-8e54-e104e95c648f', '8aeb44e7-a63d-5ec5-88ad-7703ff6b0cc4', '29b4a375-280e-5939-8577-faff77c5129c')
Preload `vulnerabilities`
SELECT
"vulnerabilities".*
FROM
"vulnerabilities"
WHERE
"vulnerabilities"."id" IN (545, 544, 535, 543, 542, 541, 540, 539, 538)
The only query change introduced in this MR is removing a column from ordering.
Testing locally
- Checkout this branch
- Import the following project: https://gitlab.com/gitlab-org/govern/demos/sandbox/minac/static-security-reports
- Run a pipeline
- Go to the pipeline security tab
- Verify the page is working correctly
- Enable the feature flag
- Go to the same pipeline security tab
- Verify the page is working correctly
- Additionally you can view the API response
MR acceptance checklist
This checklist encourages us to confirm any changes have been analyzed to reduce risks in quality, performance, reliability, security, and maintainability.
-
I have evaluated the MR acceptance checklist for this MR.