Enqueue build hooks with a build id instead of the payload

What this does

Ci::Build#execute_hooks now enqueues the build id and a small state snapshot instead of the full webhook payload:

Ci::ExecuteBuildHooksWorker.perform_async(project.id, id, hook_state_attrs)

Ci::Build#build_data is removed, because this was its only caller.

The worker gained the ability to accept this shape in 19.4 via !254541 (merged). This is the second of three steps and the one that actually reduces the queue footprint.

Why

During INC-13984 the concurrency limit for this worker deferred about 2.86 million jobs into sidekiq:concurrency_limit:throttled_jobs:{ci/execute_build_hooks_worker}. That list has no TTL, and MEMORY USAGE on the key reported 9.36 GB of a 10.3 GB dataset on shard 04 of redis-cluster-shared-state, which runs noeviction. Each deferred job carried a full Gitlab::DataBuilder::Build serialization, roughly 3.3 KB.

With an id and a snapshot of scalars, each entry is far smaller. The resulting list size has not been measured.

The state snapshot

Fields describing build state at the moment the hook fired are passed rather than re-read, because a throttled job can run long after the transition that queued it: build_status, build_started_at, build_finished_at, build_duration, build_queued_duration and build_failure_reason.

The ISO variants are not passed. The worker derives build_started_at_iso and build_finished_at_iso from whichever timestamps are pinned, so the two cannot disagree.

Two fields still reflect state at execution time rather than at the transition: runner and retries_count. A hook queued for a created build that runs after the build started will report the assigned runner where it previously reported none.

Tradeoff: Gitaly

DataBuilder::Build reads the pipeline's commit. Ci::Pipeline#commit is Commit.lazy, which resolves through Repository#commits_by to Gitlab::Git::Commit.batch_by_oid (app/models/repository.rb:139). That method is not cached.

Verified: each worker run makes one Gitaly commit lookup. BatchLoader does not batch across jobs.

Not established: whether that lookup is additional. If the commit was already memoized on the pipeline instance at transition time, this adds one call per hook. If it was not, the call only moves. The three execute_hooks call sites have not been traced.

So the delta is zero to one Gitaly lookup per hook, and it is unmeasured. Worth watching Gitaly commit read rate for this worker after deploy.

Behavior change

If the build is deleted before a throttled job runs, the hook is now dropped. Previously the payload traveled with the job, so it would still be delivered. This matches the worker's existing return unless project and is covered by a spec.

This is not a root cause fix

The payload size explains why the backlog reached 9.36 GB. It does not explain why the backlog formed. Two things remain open:

• What caused the arrival rate to exceed the drain rate. The concurrency limit was flat at 1500 for the whole week, so the change was in arrivals or in per-job duration.
• Why SidekiqConcurrencyLimitQueueBacklogged did not fire. It alerts above 100k jobs held for an hour, and this reached 2.86 million.

The throttle list also has no TTL and no size cap, so any throttled worker with a large argument can repeat this.

Testing

spec/models/ci/build_spec.rb #execute_hooks — 3 examples, 0 failures. spec/workers/ci/execute_build_hooks_worker_spec.rb — 16 examples, 0 failures.