Index vulnerability due dates in Elasticsearch
What does this MR do and why?
First of four stacked MRs for the Vulnerability Report Due date filter. This one adds a due_date field to the vulnerability Elasticsearch indexes. Nothing queries it yet.
New Vulnerability Report filters are ES-only. Due dates live in vulnerability_finding_due_dates (one row per finding), not on vulnerability_reads, so the value has to be looked up at index time:
due_date: { type: 'date' }is added to the type mapping, with a mapping migration for each index:vulnerabilitiesand the dual-writtenvulnerability_reads.- A
DueDatepreloader for each index reads the finding's due date in one query per batch. It is registered in bothEnhancedProxyclasses. due_dateis added toFIELDS_WITH_MIGRATIONSandPRELOADED_FIELDS, with a new schema version for each index (26_08and26_06). The field is only written once its mapping migration has finished.
This mirrors remediated_at (b7ecff6b) for the mapping and latest_flag (e1ce4d8d) for the preloaders.
Stack (each MR targets the one before it):
- This MR: ES field and preloaders
- !266732 (merged): reindex when a due date changes, plus the backfill
- !266733 (merged): ES query and GraphQL arguments, behind
vulnerability_due_date_filter - !266734: frontend filter token
Notes for reviewers
20261025120000_add_ascp_security_context_*mapping migrations are already on master, but their write path isn't. If that lands first, it may take schema version26_08, and this MR's version needs to move up.- All specs pass locally against Elasticsearch 9.4.4, including the two
:elasticmapping-migration specs.
References
- Epic: https://gitlab.com/groups/gitlab-org/-/work_items/20009 (Filter for 'Due Dates' on Vulnerability Report)
- Closes #638047
- Parent epic: gitlab-org#19275
- Due date data model: #592222 (closed)
Screenshots or screen recordings
Not applicable: no UI change. Verified in GDK against Elasticsearch 9.4.4: the vulnerabilities index mapping has due_date: { type: 'date' }, and after indexing, the six gitlab-org/gitlab vulnerabilities with a due date have it in their documents.
How to set up and validate locally
Requires Elasticsearch in GDK with advanced vulnerability management.
-
Run the ES migrations, then check the mapping:
curl -s localhost:9200/gitlab-development-vulnerabilities/_mapping | jq '.[].mappings.properties.due_date' -
Set a due date on a vulnerability (vulnerability details page, or the
bulkSetVulnerabilityFindingsDueDatesmutation). -
Reindex with
bundle exec rake gitlab:elastic:index_vulnerabilities, then check that the document hasdue_date.
MR acceptance checklist
Evaluate this MR against the MR acceptance checklist. It helps you analyze changes to reduce risks in quality, performance, reliability, security, and maintainability.
-
Mapping-only migrations, plus one extra batched query per indexing batch.
Both
DueDatepreloaders run this query once per indexing batch, with the batch's finding IDs:SELECT "vulnerability_finding_due_dates".* FROM "vulnerability_finding_due_dates" WHERE "vulnerability_finding_due_dates"."vulnerability_occurrence_id" IN (1, 2, 3, ...)It uses the unique index
index_vulnerability_finding_due_dates_on_occurrence_id, so it returns at most one row per finding.