Resolve "Security Dashboard: ES migration to backfill detected_at for existing vulnerabilities [BE]"

What does this MR do and why?

References

• Main issue: Security Dashboard: ES migration to backfill de... (#578071) • Charlie Kroon • 18.6
• Parent issue: Security Dashboard: Show Mean/Median Age (Vulne... (#577550) • Subashis Chakraborty, Charlie Kroon • 18.6
• Example MR: !204012 (merged)
• Docs: https://gitlab.com/gitlab-org/gitlab-development-kit/blob/main/doc/howto/elasticsearch.md#setup

How to set up and validate locally

Step 1: Before you begin

1. Make sure you have Elasticsearch running locally. You can follow this guide: https://gitlab.com/gitlab-org/gitlab-development-kit/blob/main/doc/howto/elasticsearch.md#setup

   Just a quick note: in Ruby we write SCHEMA_VERSION = 25_43, but in the Elasticsearch query you’ll use 2543

Step 2: Verify baseline (old schema)

1. In ee/lib/search/elastic/references/vulnerability.rb , make sure your schema version is:

   SCHEMA_VERSION = 25_43

2. Start Rails console (in SaaS mode):

    GITLAB_SIMULATE_SAAS=1 bundle exec rails console

3. Run bookkeeping for all vulnerabilities:

   Vulnerabilities::Read.all.each { |v| ::Elastic::ProcessBookkeepingService.track!(Search::Elastic::References::Vulnerability.new(v.vulnerability_id, "group_#{v.project.namespace.root_ancestor.id}")) }

4. Then run the processor until it shows [0, 0]. You might need to run it a few times.

   Elastic::ProcessBookkeepingService.new.execute

5. Next, in your terminal (not Rails console), check that the old schema docs exist (the count should be > 0):

   curl -X GET "localhost:9200/gitlab-development-vulnerabilities/_count" -H 'Content-Type: application/json' -d'
    {
      "query": {
        "term": {
      "schema_version": 2543
      }
     }
   }'

Step 3: Bump schema and reindex

1. Now, go back to ee/lib/search/elastic/references/vulnerability.rb , and change your schema version to:

      SCHEMA_VERSION = 25_44 

2. Restart background jobs and open a fresh Rails console:

     gdk restart rails-background-jobs
     GITLAB_SIMULATE_SAAS=1 bundle exec rails console

3. Run the migration and initial bookkeeping until it shows [0, 0] (you might need to run it a few times):

    Elastic::DataMigrationService[20251024110346].migrate
    ::Elastic::ProcessInitialBookkeepingService.new.execute

Step 4: Validate results

1. Now let’s make sure everything worked as expected. First, check that old schema records are gone (count should be 0):

   GET /gitlab-development-vulnerabilities/_count
   {
     "query": {
       "term": {
         "schema_version": 2543 # old schema
       }
     }
   }                         

   You should see something like:

   `{"count":0,"_shards":{"total":5,"successful":5,"skipped":0,"failed":0}}`

2. Then confirm the new schema shows data (count > 0):

   GET /gitlab-development-vulnerabilities/_count
   {
     "query": {
       "term": {
         "schema_version": 2544 # new schema
       }
     }
   }

   Example result:

   `{"count":3154,"_shards":{"total":5,"successful":5,"skipped":0,"failed":0}}%`

3. As a final check, make sure the migration completed successfully:

   Elastic::DataMigrationService[20251024110346].completed?

   This should return true .

4. Done! 😀 🎉

MR acceptance checklist

Evaluate this MR against the MR acceptance checklist. It helps you analyze changes to reduce risks in quality, performance, reliability, security, and maintainability.

Related to #578071