Accept a build id and state snapshot in ExecuteBuildHooksWorker

What this does

Ci::ExecuteBuildHooksWorker#perform will now accepts build id plus a small state snapshot as an alternative to the full webhook payload:

perform(project_id, build_data_or_build_id, build_state = nil)

When the second argument is a Hash -> it behaves exactly as before.

When it is an id -> the worker loads the build, rebuilds the payload with Gitlab::DataBuilder::Build, and applies the snapshot over it.

Notes:

Changing worker arguments has to span milestones, per the Sidekiq compatibility guidelines. During a deploy, web nodes would start enqueueing the new argument shape before every Sidekiq node can parse it. The plan, from @avielle's comment: !254541 (comment 3811514090):

  • 19.4, this merge request: the worker accepts the new arguments. The caller does not use them.
  • 19.5: execute_hooks switches to passing the build id and snapshot.
  • 19.6, optional: remove the is_a?(Hash) branch once no old-shape jobs remain.

Background

During INC-13984 the concurrency limit for this worker deferred about 2.86 million jobs into sidekiq:concurrency_limit:throttled_jobs:{ci/execute_build_hooks_worker}. That list has no TTL, and MEMORY USAGE on the key reported 9.36 GB of a 10.3 GB dataset on shard 04 of redis-cluster-shared-state, which runs noeviction. Each deferred job held a full Gitlab::DataBuilder::Build serialization, roughly 3.3 KB.

The reduction in queue size arrives with the 19.5 change, not with this one. The concurrency limit cannot safely be restored until then.

Why the snapshot exists

Fields that describe build state at the moment the hook fired can change before a throttled job runs, so they are passed rather than re-read: status, started_at, finished_at, their ISO variants, duration, queued_duration and failure_reason. Raised by @avielle in !254541 (comment 3810555767).

Edited by Laura Montemayor

Merge request reports

Loading
Loading