Filter dependency scanning findings by dependency change
What does this MR do and why?
Applies the dependency_change and dependency_type scanner settings when evaluating merge request approval policies, as part of the stack for #635482.
GroupedFindingsEvaluatortakes an optionaldependency_filter. For scanner groups that setdependency_changeordependency_type, it filters the findings throughDependencyFilterinstead of returning all context-unaware UUIDs.UpdateApprovalsServicebuilds the filter from the source pipeline and the target pipeline of the rule, and passes it for the source pipeline evaluation only.Security::Finding.context_unaware_uuids_with_locationplucks each context-unaware UUID together withfinding_data->'location', which the filter needs.Rule#has_scanner_overrides?now compares onlydependency_changeanddependency_typewithany, the only attributes that accept that value.- The docs describe the
dependency_changeanddependency_typescanner settings, from !267119 (merged), which was merged into this branch. - Only the merge request pipeline's findings are filtered. Target findings are still subtracted unfiltered, so the newly detected baseline doesn't change. For previously existing states, a filtered scanner counts only the merge request pipeline's findings.
The settings reach evaluation only when approval_policy_dependency_scanning_filters is enabled: with the flag disabled, ApprovalPolicySource#scanner_configurations (from the first MR in the stack) drops them, so rules follow today's evaluation path. The flag is disabled by default.
Approach: the plan was reviewed and agreed on the issue (revision 2): #635482 (comment 4070943000)
Database
The new scope adds finding_data->'location' to the SELECT DISTINCT that distinct_context_unaware_uuids already runs. It only runs for scanner groups that set dependency_change or dependency_type, and only when approval_policy_dependency_scanning_filters is enabled. The security_findings scan, the partition and severity filters and the dismissed check are unchanged, so the diff is one extra selected column.
Measured on a production-size sec clone for a pipeline with 343 matching dependency scanning findings (critical and high). Both queries return the same 343 rows before and after. Cold means right after a clone reset.
1. Related pipelines (related_pipeline_ids present)
SELECT DISTINCT COALESCE(context_unaware_uuid, uuid), finding_data->'location'
FROM "security_findings"
INNER JOIN "security_scans" ON "security_scans"."id" = "security_findings"."scan_id"
WHERE "security_scans"."project_id" = <project_id>
AND "security_scans"."pipeline_id" IN (<pipeline_id>)
AND "security_scans"."latest" = TRUE
AND "security_scans"."status" = 1
AND "security_findings"."severity" IN (7, 6)
AND "security_scans"."scan_type" = 2
AND (NOT EXISTS (
SELECT 1 FROM "vulnerabilities"
INNER JOIN "vulnerability_occurrences" ON "vulnerability_occurrences"."vulnerability_id" = "vulnerabilities"."id"
WHERE "vulnerabilities"."state" = 2 AND (vulnerability_occurrences.uuid = security_findings.uuid)))The plan is an index scan on security_scans by pipeline, then an index scan on the scan_id and severity index of each security_findings partition, then an anti join with one index lookup per finding.
| Before (uuid only) | After (uuid and location) | |
|---|---|---|
| Cold | 529 ms, 4454 hit / 4874 read | 604 ms, 4975 hit / 4929 read |
| Warm | 6.9 ms, 3727 hit / 0 read | 12.3 ms, 4299 hit / 0 read |
2. Source pipeline only (no related pipelines)
SELECT DISTINCT COALESCE(context_unaware_uuid, uuid), finding_data->'location'
FROM "security_findings"
INNER JOIN "security_scans" ON "security_findings"."scan_id" = "security_scans"."id"
INNER JOIN "security_scans" "scans_security_findings" ON "scans_security_findings"."id" = "security_findings"."scan_id"
WHERE "security_scans"."pipeline_id" = <pipeline_id>
AND "security_findings"."partition_number" = <partition_number>
AND "security_scans"."latest" = TRUE
AND "security_scans"."status" = 1
AND "security_findings"."severity" IN (7, 6)
AND "security_scans"."scan_type" = 2
AND (NOT EXISTS (
SELECT 1 FROM "vulnerabilities"
INNER JOIN "vulnerability_occurrences" ON "vulnerability_occurrences"."vulnerability_id" = "vulnerabilities"."id"
WHERE "vulnerabilities"."state" = 2 AND (vulnerability_occurrences.uuid = security_findings.uuid)))The plan reads a single partition through the same index, then the same anti join.
| Before (uuid only) | After (uuid and location) | |
|---|---|---|
| Cold | 494 ms, 4451 hit / 4755 read | 521 ms, 4972 hit / 4810 read |
Reading finding_data adds about 520 buffer hits and 55 reads for 343 rows. Most of the time in both versions is the per-finding anti join that already exists, and cold timings vary with clone I/O.
3. Large pipelines
In a sample of 18,562 dependency scanning pipelines (two hours of data on the clone), the median pipeline has 8 findings, the 99th percentile 846, and 0.7% have 1,000 or more. Counting only critical and high, the median is 2, the 99th percentile 390, and 0.3% have 1,000 or more. The largest has 14,642 critical and high findings.
Query 1 on the largest pipeline and on one with 5,643 critical and high findings. The plan has the same shape as above, and the result is sorted in memory (3.2 MB for 14,642 rows, against 1.8 MB before).
| Findings | Before (uuid only) | After (uuid and location) | |
|---|---|---|---|
| 14,642 | Warm | 145 ms, 144,662 hit | 231 ms, 154,931 hit |
| 14,642 | Cold | 34.3 s, 42,495 read | 3.9 s, 43,411 read |
| 5,643 | Warm | 68 ms, 56,233 hit | 125 ms, 69,858 hit |
| 5,643 | Cold | 11.4 s, 18,261 read | 2.2 s, 19,472 read |
Warm, selecting the location adds about 60 to 90 ms for these sizes. Cold timings are not comparable between runs: the same query on the 5,643-finding pipeline took 1.7 s before and 2.2 s after in a rerun with the order swapped. Cold reads differ by 2 to 7%, so I compare reads instead.
References
- Closes #635482
- Feature flag rollout: https://gitlab.com/gitlab-org/gitlab/-/work_items/635635
- Follow-ups:
- Rules with only previously existing states: https://gitlab.com/gitlab-org/gitlab/-/work_items/635632
- Rule builder support: #635633
- Container scanning: #635634
Screenshots or screen recordings
Not applicable: backend only.
How to set up and validate locally
These steps need a GDK runner that can run dependency scanning.
- Enable the flag in the Rails console:
Feature.enable(:approval_policy_dependency_scanning_filters) - In a project with a
requirements.txtand dependency scanning enabled, run a pipeline on the default branch. - Add a merge request approval policy (YAML mode):
rules: - type: scan_finding branches: [] scanners: - type: dependency_scanning dependency_change: introduced vulnerabilities_allowed: 0 severity_levels: [critical, high] vulnerability_states: [new_needs_triage] - Open a merge request that changes only application code. Findings on unchanged dependencies don't require approval.
- Change the version of a vulnerable package in
requirements.txt. The merge request requires approval. - Disable the flag and rerun the merge request pipeline. Findings on unchanged dependencies require approval again.
- Run the specs:
bundle exec rspec ee/spec/lib/security/scan_result_policies/grouped_findings_evaluator_spec.rb ee/spec/services/security/scan_result_policies/update_approvals_service_spec.rb ee/spec/lib/security/scan_result_policies/rule_spec.rb ee/spec/models/security/finding_spec.rb
MR stack
This MR is part of a stack that replaces one 760-line MR. The earlier MRs have merged, so this is the last one.
- !264451 (merged): schemas, config plumbing, and the feature flag. Merged.
- !267117 (merged) adds
DependencyFilter, with nothing referencing it yet. Merged. - This MR: !267118 (merged) wires the filter into evaluation, adds the
Security::Findingscope, and adds the docs. It targetsmaster.
MR acceptance checklist
Evaluate this MR against the MR acceptance checklist. It helps you analyze changes to reduce risks in quality, performance, reliability, security, and maintainability.