Coalesce the pipeline status subscription updates
What does this MR do and why?
This MR opts the two pipeline status subscriptions in to coalescing from !266624 (merged):
Both send the current state of one pipeline, so subscribers do not need the updates in between. That is the opt-in rule for coalescing.
| Subscription | Topic | Problem today |
|---|---|---|
ciPipelineStatusesUpdated |
one for each project, or project and commit SHA | Drops the most subscription events on GitLab.com, about 3.5 million in one week. Busy projects go over ci_pipeline_statuses_subscription for most of the working day. When the dropped event is the last one of a pipeline, the commit pipelines tab shows the wrong status until the user reloads the page. |
ciPipelineStatusUpdated |
one for each pipeline | No rate limit. Every pipeline and stage status change triggers it, because GraphqlTriggers.ci_stage_status_updated also triggers the pipeline topic. Each trigger pushes the whole pipeline to every open page of that pipeline. On GitLab.com: about 356 triggers each second on average, up to 734, and about 307 pushes to browsers each second, up to 1,662 (Grafana, 24 hours, 2026-10-21). |
What changes when graphql_subscription_coalescing is enabled for a user:
- Each websocket node runs each subscription at most once for each pipeline in each window, 500 milliseconds by default, and always runs the last update of a burst.
- The
ciPipelineStatusesUpdatedtopic is project-wide, but the merge key is the pipeline. So updates for different pipelines of a project are never merged, and every pipeline still gets its final state. - A subscriber can get an update up to one window later than today, plus the time the ActionCable worker pool needs to pick up the work.
- The failed-jobs trigger in !257495 (closed) becomes safe for large pipelines: a burst of job failures costs one push for each window, not one push for each failure.
This MR does not change the rate limit of ciPipelineStatusesUpdated. The limit still drops events before they reach the websocket nodes. Raising it to a safety value is step C2 in the work item, after the coalescer metrics look good in production.
Local verification
To check ciPipelineStatusUpdated the same way, open the pipeline page instead and filter on ciPipelineStatusUpdated.
Expected result, as recorded in GDK on this branch (10 triggers within 59 ms):
graphql_subscription_coalescing |
Messages received | Final status shown |
|---|---|---|
| Off | 10, within 72 ms | Passed |
| On | 2: the first at once, the last after 496 ms | Passed |
With the flag on, the first update of the burst runs at once. The websocket node keeps the rest and runs the newest one when the 500-millisecond window ends. The rows show Passed already in the middle of the burst, because each run loads the pipeline again and reads its current status.
Flag off on the left, flag on on the right. The panel in the lower-right corner lists the WebSocket messages the page received. The panel was added only for the recording.
Rollout
The rollout issue starts with the DRI and team members, then a percentage of users. Watch:
gitlab_graphql_subscription_coalesced_events_total{subscription=~"ci_pipeline_status(es)?_updated"}: the share ofdeferredupdates is the work saved.overflowmust stay close to 0.action_cable_graphql_coalescer_keys: memory use on the websocket nodes.action_cable_pool_pending_tasks: the ActionCable worker pool must not back up.action_cable_single_client_transmissions_total{broadcasting=~"graphql-event:ciPipelineStatus(es)?Updated"}: pushes to browsers must go down.
How to set up and validate locally
Specs
bin/rspec spec/lib/gitlab/graphql/subscriptions/action_cable_with_load_balancing_spec.rb -e "with the window that ciPipelineStatusesUpdated declares"
bin/rspec spec/graphql/subscriptions/ci/pipelinesThe first spec runs ciPipelineStatusesUpdated through the real class with its declared window. The coalescer specs in !266624 (merged) cover the rest of the behaviour.
End to end in GDK
This check sends a burst of pipeline status changes to an open page and counts what the browser receives.
-
Check out this branch and restart Puma:
gdk restart rails-web. -
Pick a pipeline you can see, for example in
gitlab-org/gitlab-test. Open the Pipelines tab of its commit:/<project>/-/commit/<sha>/pipelines. -
Open the browser developer tools. In Network, select the
/-/cableWebSocket and open Messages. Filter onciPipelineStatusesUpdated. -
In a Rails console, fire a burst of 10 status changes that ends in
success:pipeline = Ci::Pipeline.find(<id>) original = pipeline.status (['running'] * 9 + ['success']).each do |status| pipeline.update_column(:status, status) GraphqlTriggers.ci_pipeline_statuses_updated(pipeline.reload) GraphqlTriggers.ci_pipeline_status_updated(pipeline) end -
Count the
ciPipelineStatusesUpdatedmessages, and check that the pipeline row shows Passed. -
Enable the flag for your user:
Feature.enable(:graphql_subscription_coalescing, User.find_by_username('root')). Wait about a minute, because Puma caches feature flags. Reload the page, then run steps 4 and 5 again. -
Clean up:
Feature.disable(:graphql_subscription_coalescing)andpipeline.update_column(:status, original).
MR acceptance checklist
Evaluate this MR against the MR acceptance checklist.
References
- Part of https://gitlab.com/gitlab-org/gitlab/-/work_items/630854, steps C1 and C3
- Uses the coalescer from !266624 (merged)
- Replaces !266635 (closed), the separate
ciPipelineStatusUpdatedopt-in, which is merged into this MR - M0 results that ranked
ciPipelineStatusesUpdatedfirst: https://gitlab.com/gitlab-org/gitlab/-/work_items/630854 - Stale status on the commit pipelines tab, and site-wide trigger and push volume: https://gitlab.com/gitlab-org/gitlab/-/work_items/630854#note_4059560153
- Failed-jobs trigger that adds pushes to
ciPipelineStatusUpdated: !257495 (closed) - Flag rollout: #637380