Draft: Push the pipeline subscription when its failed jobs change
What does this MR do and why?
Related to #606907
Why
The MR Pipelines tab shows, for each pipeline, a badge with the number of failed jobs and an expandable list of them. Both stay current through the ciPipelineStatusUpdated GraphQL subscription. A subscriber receives a payload each time the backend triggers that subscription for the pipeline, and the payload carries whatever fields the subscriber selected, including the failed jobs count and the failed jobs connection.
Today the backend triggers it in one place: when the pipeline's own status changes. Two events change the pipeline's failed jobs without any change to the pipeline's status, so no payload is sent:
- A job fails while other jobs in the pipeline keep running. The pipeline stays
running, its failed jobs count goes up by one. - A user retries a failed job. A new attempt is created and the previous attempt is marked
retried. The pipeline may stayrunning, its failed jobs count goes down by one and the list changes membership.
The frontend has been filling this gap on the client. After the user's own retry it reads the row and the open list again. That covers the user's own click, and nothing else. A retry from another browser, from the API, or a failure reported by a runner reaches the page only on the next poll, up to 60 seconds later. It also means the frontend cannot rely on the subscription alone, which is what !257533 wants to do.
What
Trigger ciPipelineStatusUpdated for the pipeline at the two points where its failed jobs change, so the count and the list reach every subscriber the same way a status change does. The change is behind the ci_pipeline_failed_jobs_subscription_trigger feature flag.
How
Both trigger sites live in CommitStatus, the base model for builds, bridges and external statuses, so every kind of job is covered:
- A job enters
failed. The existingafter_transition any => :failedhook, which already increments a failure-reason metric, also pushes the pipeline subscription. It runs after commit, like the metric. - A retry marks older attempts as
retried.update_older_statuses_retried!is called byCi::Pipelines::AddJobServiceinside the transaction that adds the new attempt. It now counts the rows it updated and pushes the pipeline subscription only when at least one older attempt was retired, after the transaction commits. A brand new job retires nothing and sends nothing.
Why this shape and not another:
- Bounded fan-out. Both events are proportional to failures and retries, not to job transitions. A pipeline running a thousand passing jobs sends nothing extra. This matters because the per-job stage subscription rollout, #581569 (closed), was closed over cardinality concerns.
- Not the stage subscription.
ciStageUpdatedfires on every job transition and carries the job, not the pipeline. It updates an open stage dropdown, but a job payload cannot update the pipeline's count or its failed jobs connection. It is also still behindci_stage_subscriptionon self-managed instances. - Not a new subscription type. The frontend already subscribes to the pipeline and already selects the failed jobs fields it needs. The gap is on the trigger side, so the fix is on the trigger side.
- Not a richer mutation response or more client reads. Both only cover the user's own action in the user's own browser.
One thing a reviewer may stop on: ci_pipeline_status_updated has no rate limiter. The MR-wide ci_pipeline_statuses_updated topic is limited per project, but the per-pipeline topic is not, and this MR adds pushes to it. The flag exists so broadcast volume can be watched during rollout, see #630685.
How this was verified
- Model specs for both trigger sites: a dropped commit status pushes its pipeline, a retry that retires an older attempt pushes, a first attempt that retires nothing does not, and neither pushes with the flag off.
- The end-to-end behaviour is proven in !257533, which removes the frontend's client-side reads and passes the MR pipelines feature specs on this trigger alone, including a job dropped and a retry issued from outside the browser while the pipeline keeps running.
References
- Bug this enables the fix for: #606907
- Rollout issue for the flag: #630685
- Frontend fix that consumes the payloads: !257267
- Integration MR with the full design and the end-to-end specs: !257533
- Design discussion that established subscriptions alone were not enough: !253179 (comment 3801502875)
- Related fix for stale subscription payloads, rolling out: #628832 (closed)
- Stage subscription rollout, closed over cardinality: #581569 (closed)
Screenshots or screen recordings
Backend only. No UI change in this MR.
How to set up and validate locally
- Enable both flags in a Rails console:
Feature.enable(:ci_pipeline_failed_jobs_subscription_trigger) Feature.enable(:mr_pipelines_graphql) - Open an MR whose latest pipeline is running with at least one failed job and one still running. Open the Pipelines tab and keep it visible.
- In the console, drop the running job with
job.drop!. The failed jobs count on the row increases without a reload. - Retry the failed job from the API or a second browser. The count decreases and the expanded list drops the retried job, again without a reload.
- Disable the trigger flag and repeat. The row only updates on the next poll.
MR acceptance checklist
Evaluate this MR against the MR acceptance checklist.