Dual write vulnerability data to both ES indices
What does this MR do and why?
Implements dual-write support for vulnerability data to both Elasticsearch indices: the existing vulnerabilities index and the new vulnerability_reads index.
This is part of the Vulnerabilities Across Contexts (VAC) initiative. The new vulnerability_reads index uses vulnerability_reads.id as primary key (instead of vulnerability_id), which is needed to support multiple context-specific read entries per vulnerability.
Changes
Vulnerabilities::Read#elastic_reference— When thevulnerability_read_es_dual_writefeature flag is enabled and thecreate_vulnerability_reads_indexES migration has completed, returns an array of two serialized references (one for each index) instead of a single reference.Search::Elastic::Reference.serialize— HandlesArrayreturn values fromelastic_reference(pass-through).Elastic::ProcessBookkeepingService#track!— Flattens the serialized items array after mapping, so multi-ref arrays are enqueued as individual items.- Feature flag
vulnerability_read_es_dual_write— Ops flag (default off) to gate the dual-write behavior.
How it works
Vulnerabilities::Read record
→ elastic_reference
→ [Vulnerability ref string, Vulnerabilities::Read ref string]
→ ProcessBookkeepingService.track!
→ serialize + flatten
→ both refs enqueued independently in Redis
→ ProcessBookkeepingService#execute
→ each ref deserialized and indexed to its respective ES indexDependencies
This MR is chained on:
- !225631 (merged) (preloaders)
- !225822 (merged) (reference class) ← current target branch
Once those merge to master, this MR should be retargeted to master.
Feature flag
Name: vulnerability_read_es_dual_write
Type: wip
The flag is additionally gated on the create_vulnerability_reads_index ES migration having completed, to prevent writes to a non-existent index.
Closes #592362
Edited by Rushik Subba