Deduplicate scheduled code repository index jobs
Description
When Ai::ActiveContext::Code::RepositoryIndexWorker cannot get the repository lease, it schedules itself again in 10 minutes. Scheduled jobs were not deduplicated. So each push to the default branch during a long run started its own chain of retries. Each chain ended in another indexer run.
This MR adds including_scheduled: true to the deduplication settings. Ai::ActiveContext::DisableWorker uses the same settings.
- A push that arrives while a retry is scheduled is dropped. The scheduled job reads the latest commit when it runs, so it indexes that push.
- The strategy stays
until_executing. A job that is queued while a run is in progress still waits for the lease, as agreed in !255045 (merged). - A push can now wait up to 10 minutes behind a scheduled retry, also when the long run has already finished. No index run is lost.
- In normal operation, a repository has one waiting job. Chains that already exist merge at their next retry. Two chains can exist only when a job skips deduplication, for example a job deferred by the database health check.
QA
Local GDK, with the real Sidekiq client and server deduplication middleware and the real worker. The repository lease was held manually to simulate a long run. The 10-minute waits were simulated, and the indexer was stubbed to count runs.
| Scenario | Before | After |
|---|---|---|
| Long run, 6 pushes | 6 chains, 24 jobs, 6 index runs | 1 chain, 5 jobs, 1 index run |
| Push after the long run ends, retry pending | Runs at once, then 1 more run | Dropped, indexed at the next retry |
| 2 chains from before the change | 2 chains, 2 index runs | Merge at the next retry, 1 index run |
| Push during the lease retry window | 2 chains, 2 index runs | 1 chain, 1 index run |
Reference: https://docs.gitlab.com/development/sidekiq/idempotent_jobs/#scheduling-jobs-in-the-future
Related to #631257 (closed).