## Problem Statement The `Vulnerabilities::Read` model currently [overrides its primary key definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/ee/app/models/vulnerabilities/read.rb#L26) to use `vulnerability_id` instead of the actual database primary key (`id`). This override exists because the Elasticsearch index uses `vulnerability_id` as the document `_id` for syncing PostgreSQL records to Elasticsearch documents. With the **Vulnerabilities Across Contexts (VAC)** feature, we will have multiple records in the `vulnerability_reads` table for the same vulnerability across different tracked contexts (branches). This means `vulnerability_id` will no longer be unique in the table, which breaks the current ES indexing approach: 1. ES `_id` currently maps to `vulnerability_reads.vulnerability_id` 2. The same vulnerability across two branches will have the same `vulnerability_id`, causing document collisions in ES 3. We cannot rely on `vulnerability_id` as a unique identifier for ES document sync ## Proposed Solution Create a **parallel Elasticsearch index** (`VulnerabilityV2` or `VulnerabilityRead`) that uses `vulnerability_reads.id` as the document `_id`, then gracefully migrate to the new index. ### Why a parallel index? - Changing `_id` within an existing ES index is complex and risky - A parallel index allows for safe rollback if issues arise - Follows the established pattern used for the `epics`/`issues` → `workitems` migration - Avoids compatibility issues with Redis queue backlogs containing older version references ### Implementation Plan #### Phase 1: Create the Parallel Index - Create new ES reference class (e.g., `Search::Elastic::References::VulnerabilityRead` or `VulnerabilityV2`) - The new reference class will use `vulnerability_reads.id` as the document identifier instead of `vulnerability_id` - Create new ES type class with identical mappings to current `Vulnerability` type - The `Vulnerabilities::Read` model will continue to be the source of truth for ES fields - Leverage the existing preloader framework for fields without direct column mappings #### Phase 2: Begin Dual-Writing - Implement dual-write logic to index records to both old and new indices simultaneously - Dual-writing will be achieved by: - Modifying the `elastic_reference` method on `Vulnerabilities::Read` to return an array of reference strings (one for the legacy index, one for the new index) instead of a single reference string - Updating the `track` method in `Elastic::ProcessBookkeepingService` to accept arrays of references and flatten them before processing - Deletes should fail gracefully if a document is not found (to handle race conditions between indices) - **Reads remain on the old index** during this phase - Feature flag to control dual-writing: `vulnerability_read_es_dual_write` #### Phase 3: Backfill New Index - Create ES migration to backfill all existing `vulnerability_reads` records to the new index - Use `MigrationDatabaseBackfillHelper` pattern for batched processing - Ensure `vulnerability_occurrence_id` is indexed for preloads and business logic #### Phase 4: Switch Reads to New Index - Feature flag to switch search queries to the new index: `vulnerability_read_es_new_index` - Update `VulnerabilityIndexHelper` to direct reads to the new index - Update `Search::AdvancedFinders::Security::Vulnerability::SearchFinder` to use `id` as `primary_key` - Update `Search::Elastic::Preloaders::Base#record_key` to use `vulnerability_occurrence_id` for preloads - Validate data consistency between old and new indices - Monitor for any query performance regressions #### Phase 5: Cleanup - Remove dual-write logic - Remove old index via ES migration - Remove feature flags - Update documentation **Note:** Removing the `self.primary_key = :vulnerability_id` override from the `Vulnerabilities::Read` model should be addressed in a follow-up issue, as it may affect other functionality beyond Elasticsearch indexing. ## Related Issues