New REVISION of the security report schema prevents ingestion of security reports on previous 17.x GitLab releases, breaking the expected backward compatibility

Problem

A customer on 17.1 getting the following error:

[Schema] Version 15.2.1 for report type dependency_scanning is unsupported, supported versions for this report type are:
15.0.0, 15.0.1, 15.0.2, 15.0.4, 15.0.5, 15.0.6, 15.0.7, 15.1.0, 15.1.1, 15.1.2. GitLab will attempt to validate this
report against the earliest supported versions of this report type, to show all the errors but will not ingest the report

[Schema] root is missing required keys: dependency_files

The content of the security report is not ingested.

NB: the second error root is missing required keys: dependency_files should be discarded here. This error is reported due to current validation logic using the oldest vendored schema to validate a version that doesn't have any corresponding MODEL.REVISION found in the instance. In this case this led to validate the 15.2.1 report against 15.0.0, which is a report we used before GitLab 17.0, and that was indeed requiring the dependency_files property. Since 17.0 we use schema 15.1.0 which no longer require this property. This is being addressed with Validate all future-versioned security reports ... (#497415)

Root cause

TLDR: unless particular mechanisms are put in place in the Secure analyzers and CI templates, the schema version must stick to the same MODEL.REVISION for the whole year, until the next major release of GitLab. This has not been observed.

The security report schema versioning scheme is following SCHEMAVER, not SEMVER. So the versioning scheme is: MODEL-REVISION-ADDITION rather than MAJOR.MINOR.PATCH

The schema validation process in place in the GitLab rails application does not have full forward capabability, and is only able to validate newer versions that matches the same REVISION number:

A 15.2.1 compatible report can be validated against schema 15.2.0 and be ingested if valid. But if the most recent version in the instance is 15.1.x (not the same REVISION) the ingestion won't happen.

This can be confusing as in software engineering we mostly use semver, with which backward compatibility is expected up to the same MAJOR version (MODEL in schemaver). Additionally our security scanners are enabled via CI templates and the default configuration pulls the MAJOR image tag of the analyzer image. And when a new minor or patch version of an analyzer is released, the corresponding MAJOR.MINOR.PATCH docker image tag is created but it also ovverrides the existing MAJOR.MINOR and MAJOR image tags.

This means older instances will pull the new analyzer unless they manually pin the docker image tag used by the CI job.

What happened

  • In GitLab 17.0, we've published and vendored version 15.1.0 of the schema, which is what these instances will use to validate submitted reports.
  • During 17.1, 17.2, and 17.3 only the ADDITION (aka PATCH) version of the schema has been updated, ensuring old instances can still validate the newly emitted reports.
  • In GitLab 17.4 or 17.5 (TBC) the version 15.2.0 of the schema has been released which increases the REVISION version.
  • Somewhere around that time (TBC) secure analyzers have started to use this new version, sometimes transparently as they rely on a shared module to generate the report.
  • Self-managed instances of GitLab 17.1 to 17.4 (TBC) that started using these new analyzers outputting 15.2.x schema version have seen the security report rejected due to the validation mechanism only ablo to validate report with schema up to 15.1.2.

Customer Impact

Security Reports are not ingested, meaning no vulnerabilities are incorporated into the Vulnerability Management System

Possibly all customers matching these criteria are impacted:

  • self-managed versions 17.0.0 to 17.4.1
  • ultimate mostly but also premium and free for security scans available in these tiers
  • using an up to date analyzer image (default configuration) that output a report with schema 15.2.x

Affected security analyzers:

  • Composition Analysis
    • Dependency Scanning: ❌ 15.2.1
    • Container Scanning: ✅ 15.1.1 (source)
    • OCS: ✅ (do not generate a security report, upload via internal API)
  • Dynamic Analysis
    • DAST: ✅ 15.0.4
    • API Security: ✅ 15.0.0
  • Static Analysis
    • spotbugs: ✅
    • pmd-apex: ✅
    • kubesec: ✅
    • sobelow: ✅
    • GLAS: ❌ 15.2.1
  • Secret Detection: ✅ 15.1.4

Duration

  • Dependency Scanning: from 2024-09-27_08:45 UTC (Release of v5.7.1 to 2024-10-02_22:35 UTC (Release of v5.7.3
  • GLAS: from 2024-09-26_22:07 UTC (Release of v1.0.16 to 2024-10-02_22:41 UTC (Release of v1.0.17

Workaround

Customer affected by the bug can pin the impacted analyzers to an earlier patch version in the meantime.

TODO list previously working versions:

  • Dependency Scanning: 5.7.0
  • GLAS: 1.0.15
Edited by Thiago Figueiró