Fail pull mirror when Sidekiq interruptions are exhausted

Gavin Hinfeyrequested to merge
628725-mirror-worker-interruptions-exhausted into master

What?

Fail pull mirror when Sidekiq interruptions are exhausted

Why?

RepositoryUpdateMirrorWorker has retry disabled, so when Reliable Fetch quarantines a job after repeated Sidekiq restarts nothing clears the import state. It stays in started and the mirror is skipped by the scheduler until StuckProjectImportJobsWorker sweeps it 24 hours later. Retrying manually is a silent no-op during that window.

Mark the import state failed from the interruptions-exhausted callback so the normal retry backoff reschedules it. The jid check stops a late quarantine from clobbering a newer run of the same mirror.

References

Resolves #628725 (closed)

How to set up and validate locally

  1. Trigger the exhausted callback manually — In a Rails console on a GDK instance with a pull mirror configured, simulate the scenario:
project = Project.find(<id>)
import_state = project.import_state
import_state.update!(status: :started, jid: 'test-jid-123')
   
job = { 'args' => [project.id], 'jid' => 'test-jid-123' }
RepositoryUpdateMirrorWorker.interruptions_exhausted_block.call(job)
   
import_state.reload
# Expect: status == 'failed', last_error contains 'maximum number of interruptions'
# Expect: next_execution_timestamp is set (retry scheduled)
  1. Verify JID guard — Repeat with a mismatched JID ('jid' => 'wrong-jid') and confirm import_state.status remains started.
  2. Verify non-existent project — Call with a non-existent project ID and confirm no exception is raised.
  3. End-to-end (optional, harder to reproduce) — Configure a pull mirror pointing to an unreachable remote, then repeatedly restart Sidekiq while the job is running until Reliable Fetch quarantines it. Confirm the mirror transitions to failed with the expected error message rather than staying stuck in started for 24 hours.
  4. Check retry scheduling — After step 1, verify import_state.next_execution_timestamp is in the future and the mirror appears in the scheduler queue.

MR acceptance checklist

Evaluate this MR against the MR acceptance checklist. It helps you analyze changes to reduce risks in quality, performance, reliability, security, and maintainability.

Related to #628725 (closed)

Edited by Gavin Hinfey

Merge request reports