Skip to content

Use `finding_data` to generate vulnerabilities response

What does this MR do and why?

This MR introduces a new finder class to utilize the recently introduced column called finding_data instead of parsing the report artifacts for each HTTP request. We will still be parsing the report artifact to access the remediations but only the relevant parts of the artifact will be downloaded and parsed if the finding has remediations.

Summary of the changes

As this MR is a bit bigger than usual, I wanted to summarize the changes to make it easier to review. There are 3 types of changes;

1) Stop using the old Security::PipelineVulnerabilitiesFinder

This old finder class does not obey the current retention policy and keeps serving the finding records by parsing the CI job artifacts for each request after the retention period. This change could be done in a separate MR on its own but the remaining part will still be huge.

2) Introduce a new finder called Security::PureFindingsFinder

This finder class inherits from the Security::FindingsFinder to support the same filtering/sorting options. The only difference between the new finder and the parent finder is that the new finder returns Security::Findings collection which has all the data available to populate the HTTP response.

3) Switch between the old and new finders

This MR introduces a new feature flag as a safety relay to switch between the new and old finder classes.

4) Read artifact fragments to access the remediations data

As we are storing the byte offsets of each remediation object in the CI job artifact, we can now read the job artifact partially and parse those fragments to access the remediation data.

Related to Use the `finding_data` in the `Security::Findin... (#376215 - closed).

Database review

The new finder inherits from an existing one and uses exactly the same queries. Here is the list of queries;

Load `security_findings`
SELECT
    "security_findings".*
FROM
    "security_findings"
    INNER JOIN "security_scans" ON "security_findings"."scan_id" = "security_scans"."id"
    INNER JOIN "security_scans" "scans_security_findings" ON "scans_security_findings"."id" = "security_findings"."scan_id"
WHERE
    "security_scans"."pipeline_id" = 129
    AND "security_findings"."deduplicated" = TRUE
    AND "security_scans"."latest" = TRUE
    AND "security_scans"."status" = 1
    AND "security_findings"."confidence" IN (1, 2, 3, 4, 5, 6, 7)
    AND "security_scans"."scan_type" IN (1, 2, 3, 4, 5, 6, 7, 8)
    AND "security_findings"."severity" IN (1, 2, 4, 5, 6, 7)
ORDER BY
    "security_findings"."severity" DESC,
    "security_findings"."id" ASC
LIMIT 20 OFFSET 0
Preload `security_scans`
SELECT
    "security_scans".*
FROM
    "security_scans"
WHERE
    "security_scans"."id" IN (56, 55)
Preload `ci_builds`
SELECT
    "ci_builds".*
FROM
    "ci_builds"
WHERE
    "ci_builds"."type" = 'Ci::Build'
    AND "ci_builds"."id" IN (1168, 1169)
Preload `ci_job_artifacts`
SELECT
    "ci_job_artifacts"."project_id",
    "ci_job_artifacts"."file_type",
    "ci_job_artifacts"."size",
    "ci_job_artifacts"."created_at",
    "ci_job_artifacts"."updated_at",
    "ci_job_artifacts"."expire_at",
    "ci_job_artifacts"."file",
    "ci_job_artifacts"."file_store",
    "ci_job_artifacts"."file_sha256",
    "ci_job_artifacts"."file_format",
    "ci_job_artifacts"."file_location",
    "ci_job_artifacts"."id",
    "ci_job_artifacts"."job_id",
    "ci_job_artifacts"."locked",
    "ci_job_artifacts"."partition_id"
FROM
    "ci_job_artifacts"
WHERE
    "ci_job_artifacts"."job_id" IN (1168, 1169)
Preload `vulnerability_scanners`
SELECT
    "vulnerability_scanners".*
FROM
    "vulnerability_scanners"
WHERE
    "vulnerability_scanners"."id" IN (15, 12)
Preload `vulnerability_feedback`
SELECT
    "vulnerability_feedback".*
FROM
    "vulnerability_feedback"
WHERE
    "vulnerability_feedback"."finding_uuid" IN ('d4e4c243-6d59-5e75-8f95-0a0ab61b0da2', 'ee6f3d6c-481e-5002-adfd-364594e7cad2', 'f80a8203-a5ac-5699-a87e-609071041c70', 'eb78798f-058d-5665-9ccc-e4a986598a24', '2eb3ccdf-6217-56d0-a047-baab045f8a8a', '2e87cc0d-5d39-5d12-8ecc-be4dba1c38c3', 'c105de26-355f-5982-8e54-e104e95c648f', '8aeb44e7-a63d-5ec5-88ad-7703ff6b0cc4', '29b4a375-280e-5939-8577-faff77c5129c')
Preload `vulnerability_reads`
SELECT
    "vulnerability_reads".*
FROM
    "vulnerability_reads"
WHERE
    "vulnerability_reads"."uuid" IN ('d4e4c243-6d59-5e75-8f95-0a0ab61b0da2', 'ee6f3d6c-481e-5002-adfd-364594e7cad2', 'f80a8203-a5ac-5699-a87e-609071041c70', 'eb78798f-058d-5665-9ccc-e4a986598a24', '2eb3ccdf-6217-56d0-a047-baab045f8a8a', '2e87cc0d-5d39-5d12-8ecc-be4dba1c38c3', 'c105de26-355f-5982-8e54-e104e95c648f', '8aeb44e7-a63d-5ec5-88ad-7703ff6b0cc4', '29b4a375-280e-5939-8577-faff77c5129c')
Preload `vulnerabilities`
SELECT
    "vulnerabilities".*
FROM
    "vulnerabilities"
WHERE
    "vulnerabilities"."id" IN (545, 544, 535, 543, 542, 541, 540, 539, 538)

The only query change introduced in this MR is removing a column from ordering.

Testing locally

  1. Checkout this branch
  2. Import the following project: https://gitlab.com/gitlab-org/govern/demos/sandbox/minac/static-security-reports
  3. Run a pipeline
  4. Go to the pipeline security tab
  5. Verify the page is working correctly
  6. Enable the feature flag
  7. Go to the same pipeline security tab
  8. Verify the page is working correctly
  9. Additionally you can view the API response

MR acceptance checklist

This checklist encourages us to confirm any changes have been analyzed to reduce risks in quality, performance, reliability, security, and maintainability.

Edited by Mehmet Emin INAC

Merge request reports