Fall back to unknown for license_types policies on SPDX expressions

Marcos Rocharequested to merge
629625-license-types-unknown-expression-fallback into master

What does this MR do and why?

Security::ScanResultPolicies::LicenseViolationChecker (the class backing license_types policies) compares report license names literally against a policy's configured license_types names. PMDB v3 started returning real SPDX expressions for packages that previously had no classification and came back as the literal unknown sentinel, so those packages now silently escape any license_types policy that allowed or denied unknown, since the raw expression string never equals a configured name.

Behind the license_types_unknown_expression_fallback feature flag, the checker now substitutes the unknown sentinel for a report license name whenever it's a genuine multi-license SPDX expression (AND/OR/WITH), restoring the pre-v3 behavior. A single +-suffixed identifier (e.g. LGPL-2.0+) is deliberately excluded from that check: it's one license with an "or later" modifier, not a multi-license expression, and already carries a real, matchable identifier.

The licenses property already parses expressions correctly via LicenseExpressionChecker and is untouched here, as is the shared Gitlab::LicenseScanning::PackageLicenses producer both paths read from. A small shared Gitlab::SPDX::ExpressionParsing concern now holds the memoized-parse-by-name pattern both this checker and LicenseExpressionChecker need, so the two don't each carry their own copy.

Design decisions

  • Why WITH collapses to unknown here, unlike in LicenseExpressionChecker. The SPDX-aware licenses property path resolves a WITH exception by matching its base license first (Gitlab::SPDX::ExpressionEvaluator#evaluate_with). This MR intentionally does not do the same for license_types: the goal here is restoring the pre-PMDB-v3 behavior for license_types, and v2 had no base-license matching for any expression shape, WITH included. Treating WITH like AND/OR (collapse to unknown) is consistent with that goal; treating it like :plus (match the base) would be a new capability license_types never had, not a restoration.
  • Why a custom license name equal to a raw compound-expression text does not match. An earlier version of this change also allowed policy-side comparison against raw text, but this approach had a bug: when one compound expression was matched and normalized to unknown, that sentinel entered a shared comparison set, causing an unrelated compound expression elsewhere in the same report to be incorrectly swept into the match. Restricting normalization to the report side only removes this risk structurally, since each report license's match decision now depends only on its own value. The tradeoff is that custom license names matching raw text become inert; reference unknown directly in license_types policies to match compound expressions.
  • Why compound-expression dependencies collapse to unknown and get merged across distinct expressions. The license_types property predates PMDB v3's SPDX-expression support. Before v3, every unclassified package's license was the literal string unknown, with no way to see or report a more specific value. This MR restores that pre-v3 behavior intentionally: license_types should not gain any partial SPDX-expression capability, because the goal is migrating users to the newer licenses property. Treating all compound expressions (AND, OR, WITH) as unknown is consistent with that goal. As a side effect, when multiple dependencies in the same report have different compound expressions and are all denied, they get merged into one unknown entry with combined dependency names, because each license is now normalized to the same sentinel. @alan confirmed this approach in a comment on the tracking issue: https://gitlab.com/gitlab-com/request-for-help/-/work_items/5401#note_3880678670

Known limitation

  • The Reports > License compliance widget can disagree with the approval decision for compound expressions. That widget is rendered by SCA::LicenseCompliance, a separate class from LicenseViolationChecker, and evaluates a compound expression by checking each operand individually rather than falling back to unknown. A license_types policy that allows unknown correctly does not block the merge request, but the widget can still show the compound-expression license as denied. Fixing the widget is out of scope here to keep this MR's change minimal; tracked as a follow-up: #630445

How to set up and validate locally

Preconditions: none beyond the flag below.

  1. Enable the feature flag on the rails console:

    Feature.enable(:license_types_unknown_expression_fallback)

  2. Create a project.

  3. Go to Secure > Policies.

  4. Select New policy.

  5. Select Merge request approval policy.

  6. Switch to .yaml mode.

  7. Paste the following policy:

    approval_policy:
  - name: Approved licenses
    enabled: true
    rules:
      - type: license_finding
        match_on_inclusion_license: false
        license_states:
          - newly_detected
        branch_type: protected
        license_types:
          - unknown
    actions:
      - type: require_approval
        approvals_required: 1
        role_approvers:
          - developer
      - type: send_bot_message
        enabled: true

  8. Merge the merge request this creates, to activate the policy.

  9. Add an empty Gemfile.lock file to the project.

  10. Add a file called gl-sbom-gem-bundler.cdx.json reporting the license expression MIT AND Apache-2.0:

    {
    "bomFormat": "CycloneDX",
    "specVersion": "1.4",
    "serialNumber": "urn:uuid:a15e529c-2113-4a11-a694-6bc3ea4e2b53",
    "version": 1,
    "metadata": {
        "timestamp": "2022-02-23T08:02:39Z",
        "tools": [
            {
                "vendor": "GitLab",
                "name": "Gemnasium",
                "version": "2.34.0"
            }
        ],
        "authors": [
            {
                "name": "GitLab",
                "email": "support@gitlab.com"
            }
        ],
        "properties": [
            {
                "name": "gitlab:dependency_scanning:input_file:path",
                "value": "Gemfile.lock"
            },
            {
                "name": "gitlab:dependency_scanning:package_manager:name",
                "value": "bundler"
            },
            {
                "name": "gitlab:meta:schema_version",
                "value": "1"
            }
        ]
    },
    "components": [
        {
            "name": "sidekiq",
            "version": "4.2.10",
            "purl": "pkg:gem/sidekiq@4.2.10",
            "type": "library",
            "bom-ref": "pkg:gem/sidekiq@4.2.10",
            "licenses": [
                {
                    "license": {
                        "name": "MIT AND Apache-2.0"
                    }
                }
            ]
        }
    ]
}

  11. Create a new merge request adding a .gitlab-ci.yml file with the content:

    include:
  - template: Jobs/Dependency-Scanning.gitlab-ci.yml

gemnasium-dependency_scanning:
  stage: test
  script: 'pwd'
  artifacts:
    reports:
      cyclonedx: gl-sbom-gem-bundler.cdx.json

  12. Wait for the pipeline to finish.

  13. Verify the merge request is not blocked and does not require approval, because unknown is allowed and the expression fell back to it.

  14. Edit the policy to remove unknown from license_types (or deny it, by switching match_on_inclusion_license to true).

  15. Merge the resulting policy merge request, to apply the change.

  16. Verify the merge request is now blocked and requires approval, and that the security bot comment lists the dependency under unknown, proving the compound expression is treated as unknown rather than expanded or silently dropped.

  17. Negative path: edit the policy again to restore step 7's configuration (license_types: [unknown], match_on_inclusion_license: false), and merge that change.

  18. Disable the feature flag on the rails console:

    Feature.disable(:license_types_unknown_expression_fallback)

  19. Retry the pipeline on the merge request created in step 11 (Pipelines tab > Retry), so the check re-evaluates with the flag disabled. Nothing else triggers a re-check here, since the flag toggle itself changes neither the pipeline nor the policy.

  20. Verify the merge request is blocked and requires approval even though unknown is allowed, showing today's behavior (without the fix) is unchanged with the flag off.

References

Edited by Marcos Rocha

Merge request reports