Coalesce GraphQL subscription updates on the websocket nodes

What does this MR do and why?

GraphQL subscription updates come in bursts. A pipeline with hundreds of jobs changes status many times a minute, and each trigger runs the subscription again for every subscriber. Today the only protection is a rate limit, and it drops the newest events, which carry the final state. In one week, GitLab.com dropped about 3.7 million subscription events this way, and users can see a pipeline that still shows "running" after it finished.

This MR adds the second policy from https://gitlab.com/gitlab-org/gitlab/-/work_items/630854: a subscription that sends the current state of one record can declare that its updates may be merged.

class StatusesUpdated < ::Subscriptions::BaseSubscription
  coalesce
end

The default window is 500 milliseconds. A subscription can pass a longer one, such as coalesce window: 1.second.

On each websocket node, for each record:

  1. The first update runs at once and opens the window.
  2. Updates that arrive during the window are merged into one pending update.
  3. When the window ends, the pending update runs and opens the next window. When nothing is pending, the window closes.

Subscribers get at most one update for each record in each window, and they always get the last one. The update loads the record again, so it carries the current state.

In standard terms, this is a throttle with a leading and a trailing edge, per record, that keeps the newest update, like throttle in lodash or throttleTime with leading and trailing in RxJS. It is called a coalescer, not a throttle, because in this codebase "throttle" means the rate limiter, which drops events.

No subscription declares coalesce in this MR, so nothing changes yet. The next MR opts in ciPipelineStatusesUpdated. With merging in place, a later step can raise the producer rate limits to a safety value, so they almost never drop events.

Why on the websocket nodes

The work item first planned a Sidekiq debounce. It was rejected in https://gitlab.com/gitlab-org/gitlab/-/work_items/630854#note_4099485750 because each duplicate job costs WAL queries on the primary database, the debounced job bypasses the rate limit, and a duplicate that arrives while the job runs can be lost.

This MR merges in the two hooks that graphql-ruby's stream handler calls for each update, load_action_cable_message and execute_update, instead:

  • Each process receives every message for its topics, from all producers, so it can merge without any coordination.
  • GitLab has no broadcastable subscriptions, so each subscriber has its own fingerprint and its own stream handler. Each subscriber gets its own window.
  • It adds no Sidekiq jobs, no Redis calls and no database queries. It removes executions and the deliver broadcasts that come with them.
  • A deferred update goes through connection.worker_pool.async_invoke, the same path as an update from Redis. So it runs with the same ActionCable :work callbacks: request store, instrumentation, load balancing and batch loader. The scheduler thread only hands work to the pool.

Details a reviewer should check

  • Every update now takes the lazy path. load_action_cable_message returns an UnloadedMessage, and execute_update decides before it loads. This is true for every subscription, not only the ones that declare coalesce. Behavior does not change for a subscription without coalesce, but its path does:
    • execute_update parses the envelope once, and uses it both for the decision and for the fallback to the primary.
    • The loader then calls graphql-ruby's own load_action_cable_message directly, so a deferred update cannot reach the lazy override again.
    • The policy of each field is cached, so an update does not look up the schema field.
    • The code depends on the two hooks that graphql-ruby exposes for backends, on the public Event#fingerprint, and on graphql-ruby's @events map, which the deferred run reads to find the current subscribers.
  • Broadcastable subscriptions. A broadcastable subscription shares one fingerprint between subscribers. graphql-ruby calls execute_update only for the first event of a fingerprint, so only that update can open a window. No subscription is broadcastable today.
  • Merge key. Coalescer::Key is the topic, the fingerprint and the global ID of the payload. A payload that is not one record, such as { work_item_id:, action: }, is never merged. ciPipelineStatusesUpdated has a project-wide topic, but updates for different pipelines have different keys.
  • Concurrency. The coalescer keeps one Burst for each key in a Concurrent::Map. A flush drains the pending update, so the next window merges only updates that arrived after it. The cap check and the topic counter are separate steps, so the caps can go over by about the size of the worker pool.
  • WAL locations. Envelope#merge keeps the newer payload with the newest location of each database from both messages. Messages come from many processes, so the last one to arrive does not always carry the newest locations. Gitlab::Database::LoadBalancing::Lsn compares locations in Ruby, because the coalescer must not query the database. Sidekiq deduplication follows the same rule when it drops a duplicate job (DuplicateJob#update_latest_wal_location!), but it compares with pg_wal_lsn_diff on the primary. If any merged message has blank or unreadable locations, the result is blank, which reads from the primary.
  • Subscribers that leave. A deferred run looks up the current subscribers of the fingerprint with fetch, because [] on graphql-ruby's Concurrent::Map creates entries. If the subscriber left, nothing runs.
  • Memory. Each key with an open window stores one pending message, and a newer message is merged into it. The coalescer caps keys at 10,000 for each process and 1,000 for each topic. A key is one record for one subscriber, so a page that many people watch uses more keys of its topic. When a cap is full, the update runs at once, as today, and counts as overflow.
  • Failures. No error in the coalescer loses an update:
    • If a deferred update raises, the coalescer reports it and still opens the next window.
    • If the scheduler cannot open a window, the coalescer forgets the key, so the key cannot stay deferred forever.
    • If deciding to defer raises, kept_for_later? reports the error and the update runs at once.
  • Danger asks for a database review of lines with delete or update. They work on a Concurrent::Map and on the coalescer's update: argument, not on a table.

Commits

  1. The first commit adds the coalescer.

  2. The next seven come from the review in !269582 (closed). Two of them fix bugs:

    • A key could stop flushing when the scheduler raised.
    • Old updates carried into the next window, so one blank WAL location pinned every later merged update of that key to the primary.

    The other five move code out of ActionCableWithLoadBalancing into Envelope, Lsn and CoalescePolicy.

  3. Two commits cover branches that rspec:coverage found untested.

  4. The last commits come from the review threads on this MR:

    • The coalescing decision moved from a setup_stream override and a channel wrapper into load_action_cable_message and execute_update.
    • Lsn moved to Gitlab::Database::LoadBalancing, and ActionCableWithLoadBalancing uses Envelope for every broadcast.
    • A spec shows that ending a window runs no queries.
    • Each update is parsed once, and the policy of each field is cached.

Metrics

  • gitlab_graphql_subscription_coalesced_events_total{subscription, outcome}: run, deferred, flushed and overflow.
  • action_cable_graphql_coalescer_keys: records with an open window, sampled by the existing ActionCable sampler.

Feature flag

graphql_subscription_coalescing (gitlab_com_derisk, off by default). The actor is the user who subscribes, which the query context already holds, so the check runs no query. The flag can be enabled for one user or a feature group, and a percentage rollout splits viewers evenly. Signed-out subscribers have no actor, so their updates are coalesced only when the flag is enabled for everyone. Rollout: #637380

Documentation

The "Limit and coalesce subscription updates" section of the GraphQL style guide explains both policies, rate_limit and coalesce. It says when to use each one and when not to, so other teams can opt in their subscriptions.

How to set up and validate locally

bin/rspec -r spec_helper spec/lib/gitlab/graphql/subscriptions/coalescer_spec.rb \
  spec/lib/gitlab/graphql/subscriptions/action_cable_with_load_balancing_spec.rb \
  spec/graphql/subscriptions/base_subscription_spec.rb \
  spec/lib/gitlab/metrics/samplers/action_cable_sampler_spec.rb

The specs cover the window and its default, the merge of WAL locations, the key caps, subscribers that leave, the flag actor, and signed-out subscribers. spec/channels/graphql_channel_spec.rb and spec/requests/api/graphql/subscriptions also pass, so subscriptions that do not declare coalesce work as before.

Verified in GDK

With !266626 (merged) on top, which opts in ciPipelineStatusesUpdated. This check ran with a 2-second window, before the stream handler wrapper and the user actor:

  1. Open the commit pipelines tab of a commit with pipelines, so the browser subscribes to ciPipelineStatusesUpdated for the project and the commit SHA.
  2. In a Rails runner, call GraphqlTriggers.ci_pipeline_statuses_updated(pipeline) 10 times for one pipeline of that commit.
  3. Count the GraphqlChannel transmitting lines for ciPipelineStatusesUpdated in log/development.log.
graphql_subscription_coalescing How the 10 triggers arrived Results sent to the browser
Off Within 12 ms 10
On Within 12 ms 2: the first at once, then one when the window ended
On 0.5 s apart, over 4.6 s 4: the first at once, then one at each of 3 window ends

So the deferred update runs through the real ActionCable worker pool, and the window opens again while updates keep coming. The last result of each run carried the current state of the pipeline. The log had no There was an exception lines from the worker pool.

MR acceptance checklist

Evaluate this MR against the MR acceptance checklist.

References

Edited by Miguel Rincon

Merge request reports

Loading
Loading