Pull mirror stuck in started for 24h when RepositoryUpdateMirrorWorker is killed mid-run

Summary

Follow-up to #477716 (closed), which handled pull mirror jobs that die before execution starts (stranding import_state in scheduled). This covers jobs that die after starting, stranding it in started — a case the 30-minute recovery path added by #477716 (closed) does not cover, so it takes up to 24 hours to clear.

Problem

RepositoryUpdateMirrorWorker sets import_state to started, fetches, then transitions to finished/failed. Clearing started is the job's own responsibility.

For large repositories an update can run long enough to overlap with Sidekiq worker restarts (deploys, autoscaling, evictions). Each restart causes Reliable Fetcher to requeue the job; after DEFAULT_MAX_RETRIES_AFTER_INTERRUPTION = 3 interruptions it is quarantined (vendor/gems/sidekiq-reliable-fetch/lib/sidekiq/base_reliable_fetch.rb:22,180-189,266).

The project is then stuck:

  1. sidekiq_options retry: false (ee/app/workers/repository_update_mirror_worker.rb:16) — the job never runs again.
  2. import_state stays started, because fail_mirror/finish_mirror live inside the killed job.
  3. mirrors_to_sync excludes %i[scheduled started] (ee/app/models/ee/project.rb:288) — never rescheduled.
  4. fail_stuck_mirrors! matches only status: [:scheduled] (ee/app/models/ee/project.rb:299-305) — the 30-minute safety net misses it.
  5. Recovery waits for StuckProjectImportJobsWorker, gated on the job's Sidekiq status key expiring at IMPORT_JOBS_EXPIRATION = 24.hours (app/workers/gitlab/import/stuck_import_job.rb:10).

Retries cannot recover it

force_import_job! returns early when updating_mirror? is true, i.e. state is scheduled or started (ee/app/models/ee/project_import_state.rb:196-197,121-123). Update now and POST /projects/:id/mirror/pull are silent no-ops with no user feedback, and recreating the mirror config does not reset the state either — so every available remediation appears to succeed while changing nothing.

The eventual sweep logs Import timed out. Import took longer than 86400 seconds, but that figure is a hardcoded constant, not a measurement (app/workers/gitlab/import/stuck_import_job.rb:10,88-90). It implies a slow fetch when the job was in fact killed, which misdirects triage.

Proposal

Primary: handle interruption exhaustion. RepositoryImportWorker already does this (app/workers/repository_import_worker.rb:8,22-24):

include Sidekiq::InterruptionsExhausted

sidekiq_interruptions_exhausted do |job|
  new.perform_failure(job['args'].first)
end

Adding the equivalent to RepositoryUpdateMirrorWorker would mark the state failed at quarantine, triggering set_next_execution_timestamp so the scheduler retries on its next pass — avoiding the stuck state rather than shortening it.

Secondary: extend stuck_mirrors to started, keyed off last_update_started_at with its own threshold well above the longest legitimate update (the current 30-minute STUCK_JOBS_DURATION_THRESHOLD would be too aggressive). Defence in depth for processes that die without an observed interruption.

Ideas 2 and 3 in #477716 (closed) (add retries; reduce the 24-hour delay) were raised then but not implemented, and either would also have addressed this.

Related

  • #477716 (closed) — parent; fixed the scheduled half
  • #596530 — last_error not reset after successful runs, so the error shown during an incident can be stale and unrelated
Edited by 🤖 GitLab Bot 🤖