Drop stuck pages:deploy statuses when PagesWorker is exhausted

What does this MR do and why?

When the Pages Sidekiq worker is OOM-killed, the rescue StandardError in Projects::UpdatePagesService#execute never runs. Each pages:deploy GenericCommitStatus stays running, so the pipeline never finishes. retry: 3 does not help: in sidekiq-reliable-fetch an OOM kill counts as an interruption, and when interruptions run out it calls sidekiq_interruptions_exhausted, not sidekiq_retries_exhausted.

This MR adds both hooks to PagesWorker. They share one perform_failure method that:

  • Drops every running pages:deploy status on the build's pipeline with script_failure. It sets a description saying the worker was interrupted too many times or ran out of retries, and sets allow_failure the same way UpdatePagesService#error does.
  • Destroys any PagesDeployment for the build still at upload_ready: false. Using destroy means the uploader also cleans up the file.
  • Does nothing if the build no longer exists.

The change only touches the worker, which keeps it small to backport.

Tradeoff: Statuses aren't linked to a build, and each retry creates a new one, so we drop every running pages:deploy status in the pipeline. If another pages job in the same pipeline is deploying at that moment, its status is also marked failed even if its deploy later succeeds. We accept this because the alternative is a pipeline that never finishes.

References

Related to #634781 (closed)

Screenshots or screen recordings

Not applicable (backend only).

How to set up and validate locally

In a Rails console, with a pipeline pipeline that has a pages build build:

stage = pipeline.stages.find_or_create_by!(name: 'deploy', position: GenericCommitStatus::EXTERNAL_STAGE_IDX, project: pipeline.project)
s = GenericCommitStatus.create!(pipeline: pipeline, project: pipeline.project, ci_stage: stage, name: 'pages:deploy', status: :running, stage_idx: stage.position)
PagesWorker.interruptions_exhausted_block.call({ 'args' => ['deploy', build.id] })
s.reload.status # => "failed"

MR acceptance checklist

Evaluate this MR against the MR acceptance checklist. It helps you analyze changes to reduce risks in quality, performance, reliability, security, and maintainability.

Edited by Naman Jagdish Gala

Merge request reports

Loading
Loading