Deduplicate scheduled code repository index jobs

Description

When Ai::ActiveContext::Code::RepositoryIndexWorker cannot get the repository lease, it schedules itself again in 10 minutes. Scheduled jobs were not deduplicated. So each push to the default branch during a long run started its own chain of retries. Each chain ended in another indexer run.

This MR adds including_scheduled: true to the deduplication settings. Ai::ActiveContext::DisableWorker uses the same settings.

  • A push that arrives while a retry is scheduled is dropped. The scheduled job reads the latest commit when it runs, so it indexes that push.
  • The strategy stays until_executing. A job that is queued while a run is in progress still waits for the lease, as agreed in !255045 (merged).
  • A push can now wait up to 10 minutes behind a scheduled retry, also when the long run has already finished. No index run is lost.
  • In normal operation, a repository has one waiting job. Chains that already exist merge at their next retry. Two chains can exist only when a job skips deduplication, for example a job deferred by the database health check.

QA

Local GDK, with the real Sidekiq client and server deduplication middleware and the real worker. The repository lease was held manually to simulate a long run. The 10-minute waits were simulated, and the indexer was stubbed to count runs.

Scenario Before After
Long run, 6 pushes 6 chains, 24 jobs, 6 index runs 1 chain, 5 jobs, 1 index run
Push after the long run ends, retry pending Runs at once, then 1 more run Dropped, indexed at the next retry
2 chains from before the change 2 chains, 2 index runs Merge at the next retry, 1 index run
Push during the lease retry window 2 chains, 2 index runs 1 chain, 1 index run

Reference: https://docs.gitlab.com/development/sidekiq/idempotent_jobs/#scheduling-jobs-in-the-future

Related to #631257 (closed).

Edited by Arturo Herrero

Merge request reports

Loading
Loading