Skip to content

Update `vulnerability_reads` scanner in the ingestion pipeline

What does this MR do and why?

As described in the linked issue, the scanner data can be inconsistent between the vulnerability_reads and vulnerability_occurrences tables for the same vulnerability.

This should never be the case, as they should always be referring to the same data. This is because vulnerability_reads is just a de-normalized read-optimized table containing heavily read data from the vulnerability and vulnerability_finding models

The scanner reference is being properly updated for the finding in the finding_map class.

This change makes it so that the existing vulnerability_reads is also updated with a new scanner reference

MR acceptance checklist

Please evaluate this MR against the MR acceptance checklist. It helps you analyze changes to reduce risks in quality, performance, reliability, security, and maintainability.

Screenshots or screen recordings

Before After
before after

How to set up and validate locally

prerequisites

  • a EE enabled local gdk
  • ci runners running locally
  • clone the example repo

without the MR patch

  1. set the reports/sast.json file in the example project to this:

    example test vulnerability
    {
        "schema": "https://gitlab.com/gitlab-org/security-products/security-report-schemas/-/raw/v14.0.0/dist/sast-report-format.json",
        "version": "15.0.0",
        "scan": {
          "start_time": "2020-08-24T07:30:25",
          "end_time": "2020-08-24T07:54:47",
          "status": "success",
          "type": "sast",
          "scanner": {
            "id": "minac",
            "name": "Minac manual scanner",
            "url": "https://gitlab.com/minac",
            "version": "0.0.1",
            "vendor": {
              "name": "Minac"
            }
          },
          "analyzer": {
            "id": "minac",
            "name": "Minac manual analyzer",
            "version": "0.0.1",
            "vendor": {
              "name": "Minac"
            }
          }
        },
        "vulnerabilities": [
          {
            "message": "Test vulnerability - 0",
            "category": "sast",
            "description": "Test vulnerability description",
            "id": "d1b90d3d-607b-4b92-a68a-552f7ba83deb",
            "cve": "d1b90d3d-607b-4b92-a68a-552f7ba83deb",
            "severity": "Critical",
            "identifiers": [
              {
                "name": "Test CVE identifier",
                "type": "CVE",
                "value": "CVE-2023-XXXX"
              },
              {
                "name": "CWE identifier",
                "type": "CWE",
                "value": "CWE-79"
              }
            ],
            "location": {
              "file": "README.md",
              "start_line": 0
            },
            "scanner": {
              "id": "minac",
              "name": "Minac"
            },
            "solution": "No solution"
          }
        ],
        "remediations": []
    }
  2. run the pipeline for the main branch

    • this will seed a vulnerability that has a scanner external_id of "minac"
  3. edit the reports/sast.json to have a new scanner id image

  4. run a pipeline again

  5. find the vulnerability in the rails console

  6. compare vulnerability.finding.scanner.external_id and vulnerability.vulnerability_read.scanner.external_id

    # find the vulnerability we seeded
    vulnerability = Vulnerability.find(541)
    
    # compare the scanner references. Without the patch, they will be
    # different. With the patch, they will match
    
    vulnerability.finding.scanner.external_id
    # => "minac-new-version"
    
    vulnerability.vulnerability_read.scanner.external_id
    # => "minac"

with the MR patch

  1. follow the same steps as above, though this time you should see the vulnerability.vulnerability_read.scanner and vulnerability.finding.scanner stay in sync whenever you edit the scanner id in reports/sast.json

Related to #442929 (closed)


Changelog: fixed
EE: true

Edited by Michael Becker

Merge request reports