Coalesce the pipeline status subscription updates

What does this MR do and why?

This MR opts the two pipeline status subscriptions in to coalescing from !266624 (merged):

Both send the current state of one pipeline, so subscribers do not need the updates in between. That is the opt-in rule for coalescing.

Subscription Topic Problem today
ciPipelineStatusesUpdated one for each project, or project and commit SHA Drops the most subscription events on GitLab.com, about 3.5 million in one week. Busy projects go over ci_pipeline_statuses_subscription for most of the working day. When the dropped event is the last one of a pipeline, the commit pipelines tab shows the wrong status until the user reloads the page.
ciPipelineStatusUpdated one for each pipeline No rate limit. Every pipeline and stage status change triggers it, because GraphqlTriggers.ci_stage_status_updated also triggers the pipeline topic. Each trigger pushes the whole pipeline to every open page of that pipeline. On GitLab.com: about 356 triggers each second on average, up to 734, and about 307 pushes to browsers each second, up to 1,662 (Grafana, 24 hours, 2026-10-21).

What changes when graphql_subscription_coalescing is enabled for a user:

  • Each websocket node runs each subscription at most once for each pipeline in each window, 500 milliseconds by default, and always runs the last update of a burst.
  • The ciPipelineStatusesUpdated topic is project-wide, but the merge key is the pipeline. So updates for different pipelines of a project are never merged, and every pipeline still gets its final state.
  • A subscriber can get an update up to one window later than today, plus the time the ActionCable worker pool needs to pick up the work.
  • The failed-jobs trigger in !257495 (closed) becomes safe for large pipelines: a burst of job failures costs one push for each window, not one push for each failure.

This MR does not change the rate limit of ciPipelineStatusesUpdated. The limit still drops events before they reach the websocket nodes. Raising it to a safety value is step C2 in the work item, after the coalescer metrics look good in production.

Local verification

To check ciPipelineStatusUpdated the same way, open the pipeline page instead and filter on ciPipelineStatusUpdated.

Expected result, as recorded in GDK on this branch (10 triggers within 59 ms):

graphql_subscription_coalescing Messages received Final status shown
Off 10, within 72 ms Passed
On 2: the first at once, the last after 496 ms Passed

With the flag on, the first update of the burst runs at once. The websocket node keeps the rest and runs the newest one when the 500-millisecond window ends. The rows show Passed already in the middle of the burst, because each run loads the pipeline again and reads its current status.

Flag off on the left, flag on on the right. The panel in the lower-right corner lists the WebSocket messages the page received. The panel was added only for the recording.

Rollout

The rollout issue starts with the DRI and team members, then a percentage of users. Watch:

  • gitlab_graphql_subscription_coalesced_events_total{subscription=~"ci_pipeline_status(es)?_updated"}: the share of deferred updates is the work saved. overflow must stay close to 0.
  • action_cable_graphql_coalescer_keys: memory use on the websocket nodes.
  • action_cable_pool_pending_tasks: the ActionCable worker pool must not back up.
  • action_cable_single_client_transmissions_total{broadcasting=~"graphql-event:ciPipelineStatus(es)?Updated"}: pushes to browsers must go down.

How to set up and validate locally

Specs

bin/rspec spec/lib/gitlab/graphql/subscriptions/action_cable_with_load_balancing_spec.rb -e "with the window that ciPipelineStatusesUpdated declares"
bin/rspec spec/graphql/subscriptions/ci/pipelines

The first spec runs ciPipelineStatusesUpdated through the real class with its declared window. The coalescer specs in !266624 (merged) cover the rest of the behaviour.

End to end in GDK

This check sends a burst of pipeline status changes to an open page and counts what the browser receives.

  1. Check out this branch and restart Puma: gdk restart rails-web.

  2. Pick a pipeline you can see, for example in gitlab-org/gitlab-test. Open the Pipelines tab of its commit: /<project>/-/commit/<sha>/pipelines.

  3. Open the browser developer tools. In Network, select the /-/cable WebSocket and open Messages. Filter on ciPipelineStatusesUpdated.

  4. In a Rails console, fire a burst of 10 status changes that ends in success:

    pipeline = Ci::Pipeline.find(<id>)
    original = pipeline.status
    (['running'] * 9 + ['success']).each do |status|
      pipeline.update_column(:status, status)
      GraphqlTriggers.ci_pipeline_statuses_updated(pipeline.reload)
      GraphqlTriggers.ci_pipeline_status_updated(pipeline)
    end
  5. Count the ciPipelineStatusesUpdated messages, and check that the pipeline row shows Passed.

  6. Enable the flag for your user: Feature.enable(:graphql_subscription_coalescing, User.find_by_username('root')). Wait about a minute, because Puma caches feature flags. Reload the page, then run steps 4 and 5 again.

  7. Clean up: Feature.disable(:graphql_subscription_coalescing) and pipeline.update_column(:status, original).

MR acceptance checklist

Evaluate this MR against the MR acceptance checklist.

References

Edited by Miguel Rincon

Merge request reports

Loading
Loading