Drop stuck pages:deploy statuses when PagesWorker is exhausted
What does this MR do and why?
When the Pages Sidekiq worker is OOM-killed, the rescue StandardError in Projects::UpdatePagesService#execute never runs. Each pages:deploy GenericCommitStatus stays running, so the pipeline never finishes. retry: 3 does not help: in sidekiq-reliable-fetch an OOM kill counts as an interruption, and when interruptions run out it calls sidekiq_interruptions_exhausted, not sidekiq_retries_exhausted.
This MR adds both hooks to PagesWorker. They share one perform_failure method that:
- Drops every
runningpages:deploystatus on the build's pipeline withscript_failure. It sets a description saying the worker was interrupted too many times or ran out of retries, and setsallow_failurethe same wayUpdatePagesService#errordoes. - Destroys any
PagesDeploymentfor the build still atupload_ready: false. Using destroy means the uploader also cleans up the file. - Does nothing if the build no longer exists.
The change only touches the worker, which keeps it small to backport.
Tradeoff: Statuses aren't linked to a build, and each retry creates a new one, so we drop every running pages:deploy status in the pipeline. If another pages job in the same pipeline is deploying at that moment, its status is also marked failed even if its deploy later succeeds. We accept this because the alternative is a pipeline that never finishes.
References
Related to #634781 (closed)
Screenshots or screen recordings
Not applicable (backend only).
How to set up and validate locally
In a Rails console, with a pipeline pipeline that has a pages build build:
stage = pipeline.stages.find_or_create_by!(name: 'deploy', position: GenericCommitStatus::EXTERNAL_STAGE_IDX, project: pipeline.project)
s = GenericCommitStatus.create!(pipeline: pipeline, project: pipeline.project, ci_stage: stage, name: 'pages:deploy', status: :running, stage_idx: stage.position)
PagesWorker.interruptions_exhausted_block.call({ 'args' => ['deploy', build.id] })
s.reload.status # => "failed"MR acceptance checklist
Evaluate this MR against the MR acceptance checklist. It helps you analyze changes to reduce risks in quality, performance, reliability, security, and maintainability.