Keep WAL locations when deferring Sidekiq jobs

What does this MR do and why?

Gitlab::SidekiqMiddleware::SkipJobs defers a job by pushing it again with perform_in(delay, *job['args']). The new job has no WAL locations, so LoadBalancing::SidekiqClientMiddleware records the deferring session's instead. SkipJobs runs before LoadBalancing::SidekiqServerMiddleware, so that session reads from a replica, and the job ends up with a replica's location, which can be older than the data it was queued for.

When the deferred job runs, the middleware checks replicas against that older location. A replica that still lags behind the job's data passes the check, so a :delayed or :sticky job can read stale rows. This needs replica lag longer than the deferral delay (5 seconds or more). That's uncommon on its own, but deferrals happen when database health signals stop, which is also when replicas lag, so the two often coincide (see the measurements below).

Deferral now keeps the job's wal_locations, wal_location_sources and dedup_wal_locations, as it already keeps its call-chain keys. Concurrency limit resumes already restore the original locations, so deferral now behaves the same.

Why it matters for advanced search

!264026 (merged) moves Search::Elastic::IndexBulkCronWorker to data_consistency :sticky. Each job only indexes refs that committed before its WAL location, so a deferred job checked against an older replica location could index a stale row. The document then stays stale until the record changes again. On GitLab.com, IndexBulkCronWorker was deferred by the database health check about 110,000 times in the 7 days to 2026-11-05.

Measurements (gprd, 7 days to 2026-11-05)

Replica lag from pg_replication_lag, read replicas only:

Cluster Minutes with any replica lagging >5 s Typical replica >5 s Max lag
main (14 replicas) 129 (1.28%), in 96 episodes 0.20% 40 s
ci 0.16% 0.01% 11 s
sec 0.03% 0.01% 34 s
  • Every main replica lags more than 5 s about 0.2–0.45% of the time, and the spikes stay below the load balancer's default 60-second max_replication_lag_time, so these replicas keep serving reads. 75 of the 129 minutes were between 13:00 and 19:00 UTC.
  • Deferrals happen at the same times. The 110,469 IndexBulkCronWorker deferrals fell in only 3,680 of the 10,080 minutes, and 4,076 of them ran in a minute when a main replica lagged more than 5 s.
  • Weighting each by the share of main replicas lagging at the time, about 980 deferred jobs a week would read from a replica lagging more than 5 s. This MR makes those jobs check replicas against the WAL location they were queued with, so a lagging replica no longer passes.
Out of scope: a related gap in concurrency limit resumes

Concurrency limit resumes restore wal_locations, and DuplicateJobs::Server sets dedup_wal_locations again from the duplicate cookie when the resumed job runs, so they are covered. The remaining gap is when the cookie expires while the job is parked:

  • Cookie::UPDATE_WAL_COOKIE_SCRIPT merges a dropped duplicate's WAL locations with keepttl, so the cookie still expires duplicate_key_ttl (10 minutes by default) after the original job's check!.
  • If a duplicate is dropped just before the cookie expires and the parked job resumes just after, the job is checked only against its own older location. It can then read from a replica that hasn't replayed the duplicate's writes yet.

This needs a resume within replica-lag seconds of the expiry, so it isn't fixed here. If it's ever needed, the safe fix is on resume: when a resumed deduplicated job's cookie has expired, record the current primary WAL location instead of restoring the original. Refreshing the cookie's TTL on each dropped duplicate would also close it, but the TTL is what lets work run again when the original job is lost, so a steady stream of duplicates could then block the work indefinitely.

References

Screenshots or screen recordings

Not applicable: no UI changes.

How to set up and validate locally

Requires database load balancing with at least one replica. GDK has none by default. For this check, the primary can act as its own replica: in config/database.yml, add this under development.main, development.ci and development.sec, with your GDK's PostgreSQL socket directory:

load_balancing:
  hosts:
    - /path/to/gdk/postgresql
  1. In a Rails console, stop a :sticky worker from running, so its jobs are deferred. run_sidekiq_jobs_* flags have no YAML definition, so Feature.disable raises in development. Use the Flipper feature directly:

    Feature.get(:"run_sidekiq_jobs_Search::Elastic::IndexInitialBulkCronWorker").disable
  2. Restart Sidekiq, so it doesn't use a cached value of the flag:

    gdk restart rails-background-jobs
  3. Enqueue a job from a primary session, so it records the primary's WAL locations:

    Gitlab::Database::LoadBalancing::SessionMap.with_sessions.use_primary do
      Search::Elastic::IndexInitialBulkCronWorker.perform_async(0)
    end
  4. Once Sidekiq has picked it up and deferred it, check the deferred job:

    job = Sidekiq::ScheduledSet.new.find { |j| j.klass == 'Search::Elastic::IndexInitialBulkCronWorker' && j['deferred'] }
    job['wal_location_sources']
    # => {"main"=>"primary", "ci"=>"primary", "sec"=>"primary"}. Before this change: "replica".
  5. Remove the flag, so the worker runs again, revert the config/database.yml change, and restart Sidekiq:

    Feature.get(:"run_sidekiq_jobs_Search::Elastic::IndexInitialBulkCronWorker").remove
    gdk restart rails-background-jobs

MR acceptance checklist

Evaluate this MR against the MR acceptance checklist. It helps you analyze changes to reduce risks in quality, performance, reliability, security, and maintainability.

🤖 Generated with Claude Code

Edited by Ravi Kumar

Merge request reports

Loading
Loading