This is the follow-up item of https://gitlab.com/gitlab-org/gitlab/-/issues/322452 In https://gitlab.com/gitlab-org/gitlab/-/issues/322452 we allowed Sidekiq jobs to use read-only database replicas. Since the deduplication should always take into account the latest binary replication pointer into account, not the first one, we decided to make these two mutually exclusive, and the deduplication will not work with `sticky` or `delayed`. We should improve this, and allow using sticky and delayed on idempotent jobs as well. ### Proposed solution: * Replication strategy will store the latest WAL locations in Redis in case that the job is dropped due to the deduplication strategy. * We store this WAL location only if the job is configured to utilize LB capabilities (`data_consistency != :always`) and the write-ahead location has changed. * On job execution, the replication strategy will update job's hash with `dedup_wal_locations` containing the latest location from the Redis cache (in case it exists). * SidekiqServerMiddleware will use an updated `dedup_wal_locations` location when checking if the replica has caught up. We decided to split https://gitlab.com/gitlab-org/gitlab/-/merge_requests/66280#note_650854524 and to merge some of the changes that are by design less-risky: ### Rollout plan: #### Step1: Preparation work: - [x] Merge [Change order of sidekiq LoadBalancing middleware](gitlab!69369) - [x] Merge [Support multiple wal locations for each database](gitlab!69371) #### Step2: - [x] Merge [Allow idempotent jobs to use load balancing](gitlab!69372) - behind the ff #### Step3: Follow up work: - [x] Enable idempotency for single worker: https://gitlab.com/gitlab-org/gitlab/-/issues/341284 - [x] Add conditional check for `idempotency?` that depends on the feature flag. - [x] Rollout of [preserve_latest_wal_locations_for_idempotent_jobs feature flag](https://gitlab.com/gitlab-org/gitlab/-/issues/338350) - [x] [Remove legacy `wal locations` from SidekiqServerMiddleware](https://gitlab.com/gitlab-org/gitlab/-/issues/338213) - [x] Merge [Create Rubocop rule to set `include_shceduling: true` when enabling LB for idempotnent jobs](https://gitlab.com/gitlab-org/gitlab/-/issues/338350) - [x] [Replace hardcoded database config name :main](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/69551) when we merge unification strategy https://gitlab.com/gitlab-org/gitlab/-/issues/336566 - [x] Enable idempotency for other workers that uses Load balancing