Skip to content

Follow-up: Allow Idempotent Sidekiq jobs to use read-only database replicas

This is the follow-up item of #322452 (closed)

In #322452 (closed) we allowed Sidekiq jobs to use read-only database replicas.

Since the deduplication should always take into account the latest binary replication pointer into account, not the first one, we decided to make these two mutually exclusive, and the deduplication will not work with sticky or delayed.

We should improve this, and allow using sticky and delayed on idempotent jobs as well.

Proposed solution:

  • Replication strategy will store the latest WAL locations in Redis in case that the job is dropped due to the deduplication strategy.
  • We store this WAL location only if the job is configured to utilize LB capabilities (data_consistency != :always) and the write-ahead location has changed.
  • On job execution, the replication strategy will update job's hash with dedup_wal_locations containing the latest location from the Redis cache (in case it exists).
  • SidekiqServerMiddleware will use an updated dedup_wal_locations location when checking if the replica has caught up.

We decided to split !66280 (comment 650854524) and to merge some of the changes that are by design less-risky:

Rollout plan:

Step1: Preparation work:

Step2:

Step3: Follow up work:

Edited by Nikola Milojevic