Skip to content

Add migration to backfill token_status for vulnerabilities in ES

What does this MR do and why?

Add migration to backfill token_status in vulnerabilities

bundle exec ruby scripts/elastic-migration backfill_token_status_in_vulnerabilities

References

Issue: Backfill migration for token_status field (#572848 - closed)

Mapping MR: - !203119 (merged)

Population MR: - !204042 (merged)

Screenshots or screen recordings

Before After
25_36 Screenshot_2025-10-01_at_17.09.18 Screenshot_2025-10-01_at_17.23.07
25_37 Screenshot_2025-10-01_at_17.09.36 Screenshot_2025-10-01_at_17.23.25

How to set up and validate locally

0. Follow the ES installation instructions first.

https://gitlab.com/gitlab-org/gitlab-development-kit/-/blob/main/doc/howto/elasticsearch.md?ref_type=heads

1. Apply temporary patch

Rollback schema version in the Vulnerability ES reference:

diff --git a/ee/lib/search/elastic/references/vulnerability.rb b/ee/lib/search/elastic/references/vulnerability.rb
index 6703054034811f..1f6d0f61066238 100644
--- a/ee/lib/search/elastic/references/vulnerability.rb
+++ b/ee/lib/search/elastic/references/vulnerability.rb
@@ -7,7 +7,7 @@ class Vulnerability < Reference
         include Search::Elastic::Concerns::DatabaseReference
         include ::Gitlab::Utils::StrongMemoize
 
-        SCHEMA_VERSION = 25_37
+        SCHEMA_VERSION = 25_36
         DOC_TYPE = 'vulnerability'
         INDEX_NAME = 'vulnerabilities'
 

2. Re-index vulnerabilities at old schema

GITLAB_SIMULATE_SAAS=1 bundle exec rails console
Vulnerabilities::Read.find_each do |v|
  Elastic::ProcessBookkeepingService.track!(
    Search::Elastic::References::Vulnerability.new(
      v.vulnerability_id,
      "group_#{v.project.namespace.root_ancestor.id}"
    )
  )
end

Elastic::ProcessBookkeepingService.new.execute # run until [0,0]

3. Verify schema before migration

  • Docs on 2536 (expected >0):
curl -s "http://localhost:9200/gitlab-development-vulnerabilities/_count?pretty" \
  -H "Content-Type: application/json" \
  -d '{"query": {"term": {"schema_version": 2536}}}'
  • Docs on 2537 (expected = 0):
curl -s "http://localhost:9200/gitlab-development-vulnerabilities/_count?pretty" \
  -H "Content-Type: application/json" \
  -d '{"query": {"term": {"schema_version": 2537}}}'

4. Restore schema and run migration

  • Reset schema back to 25_37
  • Restart console and jobs:
GITLAB_SIMULATE_SAAS=1 bundle exec rails console
gdk rails-background-jobs
  • Run migration and bookkeeping:
Elastic::DataMigrationService[20251001130713].migrate
Elastic::ProcessInitialBookkeepingService.new.execute

5. Verify schema after migration

Repeat the same count queries as in step 3.

Expected:

  • All docs are on 2537
  • Count for 2536 is 0

MR acceptance checklist

Evaluate this MR against the MR acceptance checklist. It helps you analyze changes to reduce risks in quality, performance, reliability, security, and maintainability.

Edited by Ugo Nnanna Okeadu

Merge request reports

Loading