ActiveContext migration worker

What does this MR do and why?

Adds a cron worker for executing ActiveContext migrations.

It uses the Migration table to keep track of migrations. The active connection for a gitlab instance has_many migration records and we only process these (so if the connection changes, we will execute the migrations for the new connection). A record can have the following states: pending, in_progress, completed, failed.

A worker run is executed as follows:

1. Run preflight checks: return if indexing is not enabled, return if an adapter is not configured.
2. Run through the migration files and create migration records if they don't exist.
3. Check if there are failed migrations. Migrations have retries_left defaulting to 3 and we'll retry until this is zero, then stop executing the migration worker until the problem is fixed. This also means next migrations can't be run if a current migration fails.
4. Check if there's a migration to execute:
   1. Record is pending, in_progress or failed with retries_left > 0
5. Find the record's corresponding migration file. If it doesn't exist, return early
6. Execute the migration 3. If the migration fails, we mark as failed and decrement retries_left
7. A migration should have operations (see example).
   1. If all operations are completed, mark the migration as complete
   2. If all operations are not completed, re-enqueue the worker

-> The cron worker runs every 5 minutes and re-enqueues with a 30 second delay if the current migration is partially completed.

Once this lands on production, we'll return early because the adapter isn't configured and will see these logs:

-> The worker has comprehensive logs sent to the ActiveContext logger.

When the migration succeeds on first try:

When the migration fails and retries:

References

• #521128 (closed)

How to set up and validate locally

1. Add an initializer and create a connection
2. Add the Elasticsearch logger to the initializer

config.logger = ::Gitlab::Elasticsearch::Logger.build

1. Restart sidekiq
2. Tail the logs tail -f log/elasticsearch.log
3. Create a migration file, e.g. ee/db/active_context/migrate/20250123143913_create_merge_requests.rb

# frozen_string_literal: true

class CreateCodeEmbeddings < ActiveContext::Migration[1.0]
  milestone '17.10'

  def migrate!
    create_collection :code_embeddings, number_of_partitions: 3 do |c|
      c.bigint :project_id
      c.vector :embeddings, dimensions: 768
    end
  end
end

Happy path:

1. Either wait for the cron worker to run or execute the migration worker manually:

Ai::ActiveContext::MigrationWorker.new.perform

You should see the following logs:

When the migration fails:

1. Ai::ActiveContext::Migration.destroy_all
2. Stop elasticsearch: gdk stop elasticsearch
3. Either wait for the cron worker to run or execute the migration worker manually:

Ai::ActiveContext::MigrationWorker.new.perform

You should see the following logs:

MR acceptance checklist

Evaluate this MR against the MR acceptance checklist. It helps you analyze changes to reduce risks in quality, performance, reliability, security, and maintainability.

Related to #521128 (closed)