Bump Elasticsearch schema versions and reindex affected documents

Siddharth Dungarwalrequested to merge
bump-search-schema-versions into master

What does this MR do and why?

Summary

Bump Elasticsearch schema_version for eight search scopes. Add eight Advanced Search migrations: a database-driven backfill that re-enqueues every row for each scope. A follow-up MR will add the matching Elasticsearch-driven delete migrations that remove any document still below the new schema version.

See "Scopes not covered" below for the scopes this MR deliberately leaves out.

Why this is needed

INC-14518 caused three kinds of damage: stale documents, ghost documents (ES document, no database row), and missing documents (database row, no ES document). See "Why a database walk" below for what this MR repairs and what it leaves to the follow-up.

Schema Version Bumps

Scope Version Notes
Project 26_01 SCHEMA_VERSION constant
Merge Request 26_01 SCHEMA_VERSION constant
Milestone 26_01 SCHEMA_VERSION constant
Note 26_02 SCHEMA_VERSION constant, gated on the hidden field
Snippet 26_01 SCHEMA_VERSION constant
User 26_01 SCHEMA_VERSION constant
Group 26_02 Entry in SCHEMA_VERSIONS hash
Work Item 26_02 Replaces existing 26_01 entry

Schema version format is YYVV: two-digit year plus sequential version within that year.

Note's SCHEMA_VERSION constant only applies when hidden_field_available?; otherwise NoteInstanceProxy#schema_version falls back to PREVIOUS_SCHEMA_VERSION (25_24). On GitLab.com the gating migration add_hidden_to_notes (version 20260908174500) is already complete, so notes are stamped 26_02 there. These migrations are GitLab.com-only, so the fallback path is not reachable where they run.

Why milestones came back

A separate database-driven pass over the whole milestones table is planned in #630857, and that would cover stale and missing milestone documents on its own. But a database-driven pass can never delete ghost documents (an Elasticsearch document whose database row is gone), and that pass only runs after !257456 (merged) is deployed. Milestones are small — about 2.82 million rows on gprd — so including them here costs roughly an hour of the walk and closes the ghost gap now instead of waiting.

Migrations Added

Database backfill, using Search::Elastic::MigrationDatabaseBackfillHelper. This iterates rows from the database by primary key and enqueues each through Elastic::ProcessInitialBookkeepingService. Timestamps are unchanged from the previous version of this MR, but the class names are not — see the rename note below:

  • 20260929120000 ReindexAllProjectsFromDatabase (Project)
  • 20260929120001 ReindexAllMergeRequestsFromDatabase (MergeRequest)
  • 20260929120002 ReindexAllMilestonesFromDatabase (Milestone)
  • 20260929120003 ReindexAllSnippetsFromDatabase (Snippet)
  • 20260929120004 ReindexAllUsersFromDatabase (User)
  • 20260929120005 ReindexAllGroupsFromDatabase (Group)
  • 20260929120006 ReindexAllWorkItemsFromDatabase (WorkItem)
  • 20260929120007 ReindexAllNotesFromDatabase (Note)
Why these were renamed from `...BasedOnSchemaVersion`

These migrations don't select anything by schema version — they walk the whole table. The old name pointed readers at Search::Elastic::MigrationReindexBasedOnSchemaVersion, which isn't the helper they use; they use Search::Elastic::MigrationDatabaseBackfillHelper. The new name follows the existing ee/elastic/migrate/20240807160655_reindex_all_issues_from_database.rb, which uses the same helper. The rename happened before merge because a migration's class name is recorded in every instance's migrations index once it runs, so renaming after merge would orphan that record.

Configuration on all eight: batched!, batch_size 50_000, throttle_delay 1.minute. (50_000 matches the existing database-backfill migrations in this repo; the previous version of this MR used 9_000.)

All eight migrations:

  • Include dictionary YAML in ee/elastic/docs/
  • Include migration spec, using the shared example 'migration reindexes all data'.
  • Do not set retry_on_failure. previous_attempts is cumulative across the whole run and never resets after a successful batch, so with it set, 30 transient errors spread over a weeks-long run would halt the migration, and a halted migration blocks every later Advanced Search migration. Without it, a batched migration that raises is simply re-entered by the migration cron, so transient errors heal on their own.
  • Gated with skip_if so they only run on GitLab.com. See "Skipping self-managed and Dedicated" below.

Expected runtime

Search::Elastic::MigrationDatabaseBackfillHelper reads ceil(batch_size / 1_000) = 50 pages of 1,000 rows per worker run, and the worker is re-enqueued every throttle_delay, which is 1 minute. So the walk tops out at about 50,000 rows a minute.

Row counts below are postgres.ai estimates on gprd (rkumar555's session 61308, command 166666):

Scope gprd rows (approx.)
Merge Request 466M
Issue (backs Work Item) 134M
Project 53M
User 26M
Group 11M
Milestone 2.82M
Snippet 0.8M
Note not measured
Total ~694M

At 50,000 rows a minute, that's roughly 9.6 days for the whole backfill, and roughly 6.5 days for merge requests alone.

The notes table is the largest of the eight scopes. The migration excludes system notes, which make up most of that table, but the remaining non-system-note row count has not been measured on gprd. So the ~9.6 day figure above covers the other seven scopes only; the real backfill total is at least that and not yet known. This number should be measured before the change is scheduled.

Those row counts include rows that limited indexing filters out. The walk still pages through them, so the walk time does not shrink on an instance with limited indexing enabled. The helper's queue threshold — it stops enqueuing once the bookkeeping queue exceeds 50,000 entries — can make this slower, never faster.

Advanced Search migrations run one at a time in timestamp order, so every Advanced Search migration merged after these waits behind them for the full ~9.6 days. This is the reason the estimate matters beyond this MR: it affects other teams landing migrations, and the SRE on call if something needs attention during that window.

Skipping self-managed and Dedicated

INC-14518 was a GitLab.com incident, so no self-managed or Dedicated instance lost data, and none of them has anything to repair. Without a gate, every such instance would re-walk its own equivalent of ~694M rows for no benefit, and every later Advanced Search migration would wait behind it — days on a large instance. All eight migrations carry this line, placed right after the include and before the batched!/batch_size options:

  # INC-14518 only affected GitLab.com, so other instances have nothing to repair.
  skip_if -> { !saas_with_es? }

saas_with_es? is a class helper on Elastic::Migration (at ee/lib/elastic/migration.rb) that returns Gitlab::Saas.feature_available?(:advanced_search).

In this MR only the backfill exists; the delete migrations in the follow-up MR will carry the same gate. That matters for the same reason: the delete query matches documents whose schema_version is below the target or which have no schema_version field at all, filtered by type. An ungated delete on an instance with no backfill would match essentially every document of those eight types.

Leaving those instances on the old schema_version is harmless. Nothing gates search behaviour on these versions, and a future schema-version migration would pick the documents up.

Each dictionary YAML also gains skippable: true and a skip_condition, which ee/spec/elastic/docs_spec.rb requires of every skippable migration.

Precedent for the gate in this repo: ee/elastic/migrate/20241002103536_reindex_merge_requests_for_title_completion.rb. Each of the eight migration specs has a skip_migration? example group covering both the GitLab.com and the non-GitLab.com case.

Why a database walk

The old approach (one set of ES-driven reindex migrations) only repaired two of the three kinds of damage from INC-14518.

How each kind of damage is repaired
  • Stale documents (document in Elasticsearch, fields outdated): repaired by either approach.
  • Ghost documents (document in Elasticsearch, database row gone): the old ES-driven approach repaired these, though expensively — it re-enqueued the reference, resolved it against the database, found nothing, and issued a delete. A database walk cannot see ghost documents at all, because a database iteration never visits a row that does not exist.
  • Missing documents (database row exists, was never written to Elasticsearch): the old approach could not repair these. It selected work by querying the Elasticsearch index, so a row with no document in the index was never found. The migration would report zero documents remaining and complete successfully while the gap remained. Only a database-driven pass finds and writes them.

This MR's database walk repairs stale and missing documents. It cannot touch ghost documents, so those are left to the follow-up MR that carries the delete migrations.

Write volume is roughly the same as the old approach: bumping the schema version puts every existing document below the threshold either way.

Why the delete migrations ship separately

MigrationDatabaseBackfillHelper#completed? returns true as soon as every row has been enqueued through Elastic::ProcessInitialBookkeepingService, not once it has been indexed. Documents can still be sitting in that queue at that point.

A delete migration running then would remove those documents. conflicts: 'proceed' does not prevent this: it only skips documents modified after the delete_by_query snapshot, and a still-queued document carries the old schema_version, so it matches the delete query and is removed. It comes back only if its queued upsert later succeeds; if that upsert fails, the document stays missing until the record is next touched.

Shipping the deletes in a follow-up MR puts a human checkpoint before the destructive step. That MR is merged only after the backfills report complete on GitLab.com and the bookkeeping queue has drained.

There is precedent for shipping these separately: ee/elastic/migrate/20240807160655_reindex_all_issues_from_database.rb and its matching delete migration shipped in separate merge requests weeks apart.

Per-scope configuration

respect_limited_indexing? and item_to_preload on each backfill migration:

Scope respect_limited_indexing? item_to_preload Why
Project false { namespace: :parent } Project#maintaining_elasticsearch? only checks elasticsearch_indexing?, so limited indexing does not apply. The preload covers the root-ancestor lookup in es_parent.
MergeRequest true { target_project: { namespace: :parent } } Under limited indexing the helper calls maintaining_elasticsearch? per record, which reaches target_project.use_elasticsearch?. MergeRequest#project is an alias method for target_project, not an association, so only target_project can be preloaded.
Milestone true { project: { namespace: :parent }, group: %i[namespace_settings namespace_settings_with_ancestors_inherited_settings] } Milestone gets use_elasticsearch? from Elastic::ApplicationVersionedSearch, which delegates to project&.use_elasticsearch?. Group milestones are now indexed, and building a group milestone document reads target.group, so without the preload each one loads its group separately. The group: part matches what the Milestone.preload_for_indexing scope uses. The serializer's build_project_data reads project.self_or_ancestors_archived? and project.elastic_namespace_ancestry, so the project and its namespace parent are preloaded too.
Snippet false [] Elastic::SnippetsSearch overrides use_elasticsearch? to check only elasticsearch_indexing? (it does not consult the project — see the FIXME and #11850), so limited indexing never applies to snippets, the same as Project and User. Nothing needs preloading: the snippet's es_parent is nil.
User false [] User#maintaining_elasticsearch? only checks elasticsearch_indexing?. The user reference serializes from the id alone.
Group true :parent Matches the existing BackfillGroupsToElasticsearch migration; :parent is preloaded for the root-ancestor lookup in es_parent.
WorkItem true { project: { namespace: :parent }, namespace: :parent } Limited indexing is enabled on GitLab.com, so the helper calls maintaining_elasticsearch? per record, and WorkItem#use_elasticsearch? reads project.use_elasticsearch? when the work item has a project. Without preloading project, each page issues extra project queries across the ~134M rows. Issue#es_parent is "group_#{namespace.root_ancestor.id}", so the namespace's parent must be loaded too. The precedent, ee/elastic/migrate/20240807160655_reindex_all_issues_from_database.rb, preloaded :project.
Note true [{ project: { namespace: :parent } }, :noteable] The serializer reads noteable (polymorphic) for traversal ids, issue data and project feature access levels, and reads target.project for the archived flag and hashed root namespace id.

The shared 'migration reindexes all data' N+1 example runs with limited indexing off, so it does not exercise the per-record maintaining_elasticsearch? path described above.

Why the preload includes :parent

Namespace#root_ancestor (app/models/namespaces/traversal/linear.rb:97) only skips a query when parent_loaded_and_present?, so the namespace's parent must be preloaded too. This only covers two-level hierarchies: a project in a subgroup still costs one root-ancestor query per row, same limitation as the existing BackfillGroupsToElasticsearch migration.

Why notes exclude system notes

ReindexAllNotesFromDatabase overrides documents_after_current_id to use the existing Note.user scope (where(system: false)).

Limited indexing is enabled on GitLab.com, so MigrationDatabaseBackfillHelper does apply documents.select!(&:maintaining_elasticsearch?) per record, and Note#use_elasticsearch? is !system && super. So system notes would be dropped per record even without the scope.

The scope's purpose is therefore not correctness but cost: without it the walk pages through every system note, and system notes are the bulk of the notes table. The filter keeps those rows from being read at all. (Search::Elastic::DocumentReference#operation is database_record.present? ? index_operation : :delete, but that is not why the scope is needed.)

There is precedent for overriding documents_after_current_id on this helper: ee/elastic/migrate/20240807160655_reindex_all_issues_from_database.rb excludes epic-typed rows the same way.

This is also the first migration in the repository to walk the notes table from the database. The four existing Note migrations are either index-driven (Search::Elastic::MigrationReindexBasedOnSchemaVersion, Search::Elastic::MigrationBackfillHelper) or mapping-only (Search::Elastic::MigrationUpdateMappingsHelper).

Scopes not covered

  • blob, commit, wiki_blob — documents are fetched from Gitaly and written by gitlab-elasticsearch-indexer, not serialized from database-backed references, so they are not affected by this change.
  • Vulnerabilities — deliberately out of scope, and needs its own follow-up. An earlier version of this description claimed ee/elastic/migrate/20260908140000_backfill_remediated_at_in_vulnerabilities.rb already repaired vulnerability documents. It does not. That migration uses Search::Elastic::MigrationReindexBasedOnSchemaVersion, which only selects Elasticsearch documents whose schema_version is below 26_07. Vulnerability schema version 26_07 is gated on the add_remediated_at_to_vulnerability migration, which landed in early September 2026, before the incident, so by the time of the incident newly written vulnerability documents were already stamped 26_07 and the backfill never selects them, regardless of when the backfill itself ran. Being an Elasticsearch-driven pass, it also cannot find missing documents (a database row with no Elasticsearch document), which is precisely the gap the database-driven pass in this MR closes for the eight scopes it covers. So vulnerabilities are not repaired by anything currently merged, and repairing them needs its own backfill-and-delete pair, which is not attempted here. A follow-up issue is needed to track this.

Work Item Entry Replacement

The existing work_item 26_01 entry was replaced with 26_02 rather than adding a new entry. Two entries gating the same migration leave 26_01 unreachable: schema_version selects the highest version whose migration has completed. This also prevents flaky specs in ee/spec/lib/search/elastic/references/work_item_spec.rb:229, which asserts schema versions sort in the same order as migration timestamps.

The 26_02 entry reuses the already-landed add_title_keyword_to_work_items migration (20260911120000) rather than a new one. (20260911120001 is a different migration, backfill_title_keyword_in_work_items.) Gating a version on the migration that reindexes to it creates deadlock: the version can only be claimed once the migration completes, but the migration only completes once no documents remain below that version. Reusing a completed migration avoids this. Advanced Search migrations run in timestamp order, so add_title_keyword_to_work_items is guaranteed complete before these new migrations run.

Zero-Downtime Reindexing vs. Migrations

The database backfill in this MR is the mechanism for this change; zero-downtime reindexing (ZDR) is not an alternative.

Why migrations are required instead of ZDR

ZDR uses the Elasticsearch Reindex API to copy documents to a new index and swap the alias. Copying does not re-serialize documents. Each keeps its existing schema_version. A full ZDR would leave projects at 25_06 and work items at 25_44.

The Elasticsearch documentation (doc/integration/advanced_search/elasticsearch.md) states that ZDR should not be used to fix missing data and does not add data that is not already indexed. It also requires all Advanced Search migrations to complete before reindexing starts. The two are sequenced, not alternatives.

ZDR is a per-instance admin action and cannot ship in an MR. Self-managed instances would not converge from a release.

During ZDR, writes to the primary index pause until the alias swap. These migrations throttle instead and never pause writes.

ZDR remains the right tool for index settings or mappings that cannot be changed on a live index, such as changing a field's type or analyzer. If that becomes needed, ZDR still requires a reindex migration afterward to stamp the version.

Turning this off or limiting it

The backfill migrations re-enqueue every existing row of their type from the database, so if they put too much load on the database or Gitaly, any of the following stops or slows them. All of these run in a Rails console and need no deploy.

  1. Stop all Advanced Search migrations (bluntest; takes effect on the next cron run):

    settings = ApplicationSetting.last
settings.elastic_migration_worker_enabled = false
settings.save!

    Elastic::MigrationWorker checks this in its preflight check and returns early when false. Defaults to true, available in GitLab 18.0 and later. Note this stops every pending migration, not only the eight here.

  2. Pause all Elasticsearch indexing — the elasticsearch_pause_indexing setting, also in Admin > Settings > Search as "Pause indexing for advanced search". Broader than option 1: it also stops ordinary indexing from user activity. Use it if the cluster itself is the problem rather than these migrations.

  3. Halt just the migration that is running — this also halts anything queued behind it, since the worker processes migrations in order and stops at a halted one:

    Elastic::MigrationRecord.current_migration.halt

  4. Slow it down instead of stopping it — lower elasticsearch_max_bulk_concurrency and elasticsearch_max_bulk_size_mb, or reduce elasticsearch_requeue_workers. These govern how the bookkeeping queue is drained, which is where the database and Gitaly load originates.

Why these are the levers

Each backfill migration iterates every existing row of one scope from the database and enqueues it through Elastic::ProcessInitialBookkeepingService, which reindexes it carrying the new schema_version. Load on the database and Gitaly comes from re-serializing each row, not from Elasticsearch itself.

The backfill migrations are already throttled: each declares batched!, throttle_delay 1.minute and batch_size 50_000, and the shared helper stops enqueuing once the bookkeeping queue exceeds 50,000 entries.

batch_size and throttle_delay are class attributes on each migration (ee/app/workers/concerns/elastic/migration_options.rb), so they cannot be retuned from a console. Reducing throughput without a deploy means the settings in option 4; otherwise use options 1 to 3 to stop.

Testing

All runs were local, against a GDK Elasticsearch instance.

  • RuboCop passes on all new and changed Ruby files.
  • ee/spec/elastic/docs_spec.rb passes.
  • The eight backfill migration specs pass.
  • The notes, work items and milestones specs together: 30 examples, 0 failures, after the preload and scope changes.
  • The skip_migration? example groups cover both the GitLab.com and non-GitLab.com case for each of the eight migrations.
  • ee/spec/models/concerns/elastic/milestone_spec.rb passes.
A local-only timing artifact

Running all eight backfill specs in a single rspec process can time out locally with Elastic::TimeoutError, because each :elastic-tagged file tears down and recreates every index against a single-node cluster. The same files pass when run as separate processes, which is how CI runs them. This is a local-environment artifact, not a code problem.

References

How to set up and validate locally

  1. Enable Advanced Search in a GDK with Elasticsearch running.

  2. Index some data, for example bundle exec rake gitlab:elastic:index.

  3. Confirm the new constants are picked up:

    Elastic::Latest::ProjectInstanceProxy::SCHEMA_VERSION # => 2601
Search::Elastic::References::WorkItem::SCHEMA_VERSION # => 2602

  4. Run the migrations. Select them by version range rather than last(8), since that count breaks as soon as a later migration lands. On a GDK these are skipped by Elastic::MigrationWorker because the instance is not GitLab.com, so the steps below drive migrate directly instead of waiting for the cron.

    One migrate call on a batched backfill only enqueues a batch into the bookkeeping queue; it does not index anything. Call it repeatedly until the migration reports completed?:

    backfills = Elastic::DataMigrationService.migrations.select do |m|
  m.version >= 20260929120000 && m.version <= 20260929120007
end

backfills.each do |m|
  m.migrate until m.completed?
end

    Drain the bookkeeping queue before moving on:

    service = Elastic::ProcessInitialBookkeepingService
service.new.execute while service.queue_size > 0

    Only once the queue is drained does the next step's database-versus-Elasticsearch count comparison mean anything.

  5. Confirm the database row count for a scope matches the Elasticsearch document count for that scope. This is what catches missing documents, which is the point of the change, for example for projects:

    Project.count
Gitlab::Search::Client.new.count(index: Elastic::Latest::ProjectConfig.index_name)

    Milestones share the main index, so they need a type filter:

    Milestone.count
Gitlab::Search::Client.new.count(
  index: Elastic::Latest::Config.index_name,
  body: { query: { term: { type: 'milestone' } } }
)

MR acceptance checklist

Evaluate this MR against the MR acceptance checklist. It helps you analyze changes to reduce risks in quality, performance, reliability, security, and maintainability.

🤖 Generated with Claude Code

Edited by Siddharth Dungarwal

Merge request reports