Read incremental indexing batches from caught-up replicas

What does this MR do and why?

Search::Elastic::IndexBulkCronWorker reads every incremental indexing batch from the gitlab_main primary (data_consistency :always). During peak hours the primary is slow and a batch spends almost all its time in database queries, so the incremental queue grows during the day and only drains overnight.

Behind the search_elastic_bulk_index_replica_reads flag, the worker uses data_consistency :sticky, so the Sidekiq load balancing middleware reads the batch from a caught-up replica:

  1. :sticky alone isn't safe here. The job's WAL location is recorded when it is enqueued, but refs keep arriving in the Redis queue after that, so a replica caught up to that location can still miss them. The previous comment on the worker warned about this.
  2. Each job is therefore enqueued with a score cap: the shard's score counter, read just before the enqueue. Refs are queued after their transaction commits and the counter only grows, so every ref at or below the cap had committed before the job's WAL location. The job only processes those refs. Later refs wait for the next run, 1 second later.
  3. Both enqueues, the cron fan-out and the requeue, run in a primary session. Sidekiq records the session's WAL location, and a job that read from a replica would otherwise record that replica's location, which can be behind refs at or below the next cap.
  4. When a replica has caught up, the middleware reads from it. Otherwise it waits up to 3 × 0.5 seconds and falls back to the primary.
  5. If clear_tracking! resets the score counter after a job was enqueued, refs queued after the reset could score at or below that job's cap. The job then processes nothing, and the next cron run starts again with a fresh cap.
  6. A job without a cap always reads from the primary. During a percentage rollout, a requeue can be enqueued while the flag is off for that job, so it has no cap. Sidekiq's scheduler later pushes it through the client middleware again, with its own flag check, which can add WAL locations. Without this, the job would read an uncapped batch from a replica.
  7. The worker has defer_on_database_health_signal :gitlab_main. SkipJobs defers a job by pushing it again with the same arguments, so the deferred job keeps its cap, but on its own it would get the deferring session's WAL location, which is a replica's. !268344 (merged) keeps the job's original WAL locations, so enable the flag only after it is deployed.

With the flag off, the worker keeps :always and is enqueued without a cap. The cap is only passed while the flag is on, and the flag is off by default, so older code never receives the new argument during a deploy.

The initial queue (IndexInitialBulkCronWorker) and the retry queue are unchanged. They are already :sticky.

Why :sticky, not :delayed

With the cap, both are safe. When no replica has caught up after the wait, :delayed raises so that Sidekiq retries the job later, while :sticky reads from the primary, as the worker does today. Indexing a batch now on the primary is better than delaying it.

Trade-off: any caught-up replica

The middleware accepts a database when at least one of its replicas has caught up (databases_in_sync?). LoadBalancer#read retries a query conflict or connection error on another host without checking that it has caught up, so a retry can land on a lagging host and index a stale row. The document then stays stale until the record changes again.

On gprd, 24 hours had 0 host_query_conflict and 7 host_offline events across ~881M Sidekiq jobs. At the ~356k incremental indexing jobs of 2026-11-04, that is at most about one affected run a year. Web requests and other :sticky and :delayed jobs accept the same risk.

Rare case: changing the worker shard count

A ref's shard depends on elasticsearch_worker_number_of_shards, and queued refs stay in their old shard when it changes. So for a while, the same record can be queued in two shards, which separate jobs process under separate locks. If the older entry's job runs last, it reads from a replica caught up to its own, older WAL location, and can index an older version of the record over a newer one. The document then stays stale until the record changes again. With primary reads, both jobs read the current row.

This needs a shard count change, a second update to the same record while its first entry is still queued, and replica lag at that moment.

Measurements (gprd, 2026-11-04)
Hour (UTC) Incremental worker, primary Initial worker (:sticky), replicas
00–07 1.1–2.4 ms per record 2.1–2.4 ms per record
13–17 (peak) 52–179 ms per record 2.0–2.5 ms per record
  • At 14:00 UTC an incremental job took 182 seconds on average, 173 of them in gitlab_main queries.
  • Replicas keep the same speed at peak. The two workers index a different mix of records, so compare each column with its own overnight values.
  • Incremental batches were full (1,000 refs) in every job from 13:00 to 19:00 UTC, so the cap, which only holds back refs queued in the last second or so, shouldn't shrink them.

Sidekiq logs load_balancing_strategy for every job, which shows how often batches read from replicas during the rollout. A job without a cap can log replica but still reads from the primary, so db_main_replica_count confirms the replica reads, and db_main_replica_duration_s shows the time spent on them.

References

  • Issue: #635292
  • Rollout issue: #635285
  • Depends on !263988 (merged), which fixes the two callers that queued items before their transaction committed. It is merged and deployed to GitLab.com.
  • The flag rollout depends on !268344 (merged), which keeps WAL locations when Sidekiq defers a job. This MR can merge before it, because the flag is off by default.
  • Follow-up: #638584, a race found during review where removing a processed batch can drop a ref that was queued while the batch was being indexed. It exists with or without this MR.

Screenshots or screen recordings

Not applicable: no UI changes.

How to set up and validate locally

Requires advanced search and database load balancing with at least one replica in GDK.

  1. Enable the flag in a Rails console:

    Feature.enable(:search_elastic_bulk_index_replica_reads)
  2. Update an issue, then wait for the next Search::Elastic::IndexBulkCronWorker cron run (every minute).

  3. In gdk tail rails-background-jobs, check that the per-shard Search::Elastic::IndexBulkCronWorker jobs have two arguments, the shard number and its score cap, and log load_balancing_strategy as replica with a db_main_replica_count above 0.

MR acceptance checklist

Evaluate this MR against the MR acceptance checklist. It helps you analyze changes to reduce risks in quality, performance, reliability, security, and maintainability.

🤖 Generated with Claude Code

Edited by Ravi Kumar

Merge request reports

Loading
Loading