Add ASCP component association for findings

Harrison Petersrequested to merge
hptrs-600641-ascp-component-association into master

What & why

Associates each security finding with its ASCP (Application Security Component Profile) component so enriched vulnerability details can be surfaced. A finding maps to the component whose sub_directory is the longest leading prefix of the finding's file path.

The matching is pure GitLab-side DB work (no AI credits, does not trigger the agent). It runs asynchronously via Sidekiq rather than synchronously, because both trigger points already run in the background, the bulk re-match is unbounded work, and idempotent + retried workers give the run-to-completion guarantee. See the design discussion: https://gitlab.com/groups/gitlab-org/-/work_items/20894#note_3506768975

How it works

  • Security::Ascp::BulkSetComponentService (project:, finding_ids:) is the single write path — it matches findings and upserts/removes Vulnerabilities::AscpComponentLink rows, returning a ServiceResponse. Modeled on BulkSetDueDatesService.
  • Two workers delegate to it: UpdateAscpAssociationsBatchWorker (a batch of findings — the only worker that calls the service) and UpdateAscpAssociationsWorker (fans a whole project out into batch jobs).

Triggers (both behind the ascp_component_vulnerability_association flag, gated on the security_dashboard licensed feature)

  1. ASCP scan completes — CreateScanService enqueues a full-project re-match.
  2. New findings ingested — a dedicated ingestion task enqueues one batch job per ingestion slice (at most Security::IngestionConstants::COMPONENTS_BATCH_SIZE = 50 findings), skipped when the project has no ASCP scan.

Database review

New queries/scope introduced in Security::Ascp::BulkSetComponentService, plus the new Security::Ascp::Component.pluck_id_and_sub_directory and Vulnerabilities::AscpComponentLink#by_finding_ids scopes.

1. Security::Ascp::Scan.by_project(project_id).latest.first

SELECT "ascp_scans".* FROM "ascp_scans" WHERE "ascp_scans"."project_id" = 83 ORDER BY "ascp_scans"."scan_sequence" DESC LIMIT 1
Limit  (cost=0.29..1.49 rows=1 width=111) (actual time=0.011..0.011 rows=1 loops=1)
  Buffers: shared hit=6
  ->  Index Scan Backward using index_ascp_scans_on_project_id_and_scan_sequence on ascp_scans  (cost=0.29..6.28 rows=5 width=111) (actual time=0.010..0.011 rows=1 loops=1)
        Index Cond: (project_id = 83)
        Buffers: shared hit=6
Planning Time: 0.758 ms
Execution Time: 0.027 ms

Covered by the existing unique index index_ascp_scans_on_project_id_and_scan_sequence (project_id, scan_sequence). The ORDER BY ... LIMIT 1 is served directly by a backward index scan — no sort.

2. Component load — changed in this MR

Previously the service loaded every component of the latest scan as full ActiveRecord objects with no LIMIT.

Before:

SELECT "ascp_components".* FROM "ascp_components" WHERE "ascp_components"."scan_id" = 3
Index Scan using index_ascp_components_on_scan_id on ascp_components  (cost=0.15..21.65 rows=600 width=136) (actual time=0.009..0.050 rows=600 loops=1)
  Index Cond: (scan_id = 3)
  Buffers: shared hit=14
Planning Time: 0.371 ms
Execution Time: 0.064 ms

After (new scope) — Security::Ascp::Component.at_scan(scan_id).pluck_id_and_sub_directory(MAX_COMPONENTS + 1):

SELECT "ascp_components"."id", "ascp_components"."sub_directory" FROM "ascp_components" WHERE "ascp_components"."scan_id" = 3 LIMIT 501
Limit  (cost=0.15..18.10 rows=501 width=27) (actual time=0.003..0.055 rows=501 loops=1)
  Buffers: shared hit=9
  ->  Index Scan using index_ascp_components_on_scan_id on ascp_components  (cost=0.15..21.65 rows=600 width=27) (actual time=0.003..0.035 rows=501 loops=1)
        Index Cond: (scan_id = 3)
        Buffers: shared hit=9
Planning Time: 0.011 ms
Execution Time: 0.068 ms

Same index, but the row width drops from 136 to 27 bytes, buffers from 14 to 9, and the result is now hard-capped. Two id/sub_directory tuples replace a full AR object per component.

3. Vulnerabilities::Finding.id_in(ids).by_projects([project_id])

The service's BATCH_SIZE dropped from 1,000 to 100 in this MR, so this now runs with at most 100 ids. It previously never split at all: both callers passed either 50 or exactly 1,000 ids, so each_slice(1000) was a no-op and up to 1,000 full vulnerability_occurrences rows (including raw_metadata) were materialised at once.

SELECT "vulnerability_occurrences".* FROM "vulnerability_occurrences" WHERE "vulnerability_occurrences"."id" IN (2742, ..., 2841) AND "vulnerability_occurrences"."project_id" = 83
Index Scan using tmp_idx_vulnerability_occurrences_on_project_id_id on vulnerability_occurrences  (cost=0.29..115.03 rows=93 width=1329) (actual time=0.016..0.110 rows=100 loops=1)
  Index Cond: ((project_id = 83) AND (id = ANY ('{2742,...,2841}'::bigint[])))
  Buffers: shared hit=80
Planning Time: 2.593 ms
Execution Time: 0.136 ms

Existing Vulnerabilities::Finding scopes, reused as-is; no new index needed. The SELECT is deliberately not narrowed: Finding#file reads location['file'], and location falls back to metadata parsed from raw_metadata, so a partial select would break findings that predate the location column.

4. Vulnerabilities::AscpComponentLink.upsert_all(...) (matched findings)

INSERT INTO "vulnerability_finding_ascp_component_links" ("vulnerability_occurrence_id","ascp_component_id","project_id","created_at","updated_at")
VALUES (...)
ON CONFLICT ("vulnerability_occurrence_id") DO UPDATE SET
  ascp_component_id = EXCLUDED.ascp_component_id,
  updated_at = EXCLUDED.updated_at
WHERE "vulnerability_finding_ascp_component_links".ascp_component_id IS DISTINCT FROM EXCLUDED.ascp_component_id
RETURNING "id"
Insert on vulnerability_finding_ascp_component_links  (cost=0.00..1.75 rows=0 width=0) (actual time=0.895..0.895 rows=0 loops=1)
  Conflict Resolution: UPDATE
  Conflict Arbiter Indexes: index_vuln_finding_ascp_comp_links_on_occurrence_id
  Conflict Filter: (vulnerability_finding_ascp_component_links.ascp_component_id IS DISTINCT FROM excluded.ascp_component_id)
  Tuples Inserted: 100
  Conflicting Tuples: 0
  Buffers: shared hit=1374 dirtied=2 written=2
Planning Time: 0.139 ms
Trigger for constraint fk_8f5608019a: time=0.817 calls=100
Trigger for constraint fk_384b6ed1ca: time=1.818 calls=100
Execution Time: 3.648 ms

Conflict target is the unique index index_vuln_finding_ascp_comp_links_on_occurrence_id (vulnerability_occurrence_id). Bounded to 100 rows per statement by BATCH_SIZE. The IS DISTINCT FROM conflict filter means a re-run over unchanged links writes nothing and doesn't bump updated_at.

5. Vulnerabilities::AscpComponentLink.by_finding_ids(ids) (stale-link cleanup and the read path)

DELETE FROM "vulnerability_finding_ascp_component_links" WHERE "vulnerability_finding_ascp_component_links"."vulnerability_occurrence_id" IN (2742, ..., 2841)
Delete on vulnerability_finding_ascp_component_links  (cost=0.28..10.46 rows=0 width=0) (actual time=0.124..0.124 rows=0 loops=1)
  Buffers: shared hit=104
  ->  Index Scan using index_vuln_finding_ascp_comp_links_on_occurrence_id on vulnerability_finding_ascp_component_links  (cost=0.28..10.46 rows=100 width=6) (actual time=0.012..0.019 rows=100 loops=1)
        Index Cond: (vulnerability_occurrence_id = ANY ('{2742,...,2841}'::bigint[]))
        Buffers: shared hit=3
Planning Time: 0.541 ms
Execution Time: 0.227 ms

Covered by the existing unique index index_vuln_finding_ascp_comp_links_on_occurrence_id (vulnerability_occurrence_id) (added in the already-merged table migration).

Testing

The flag (ascp_component_vulnerability_association) is beta/off-by-default and gated on the security_dashboard licensed feature, so it isn't reachable through any UI yet. To exercise it in a local GDK:

  1. Clone the fixture project https://gitlab.com/hptrs/ascp-component-association-test (private; ping me if you need access) into your GDK — its TESTING.md has copy-pasteable console steps.
  2. It walks through: enabling the flag, seeding an Ultimate license, creating an Security::Ascp::Scan + two Security::Ascp::Components whose sub_directory matches the repo's app/services/auth / app/models folders, creating findings pointing at files in/out of those directories, running UpdateAscpAssociationsWorker inline, and verifying the resulting Vulnerabilities::AscpComponentLink rows (including the unmatched-finding and flag/license-off cases).
  3. The ascpScanCreate / ascpComponentCreate GraphQL mutations also work against this project if you'd rather drive it through GraphiQL instead of the console.

References

Edited by Harrison Peters

Merge request reports