Coalesce GraphQL subscription updates on the websocket nodes
What does this MR do and why?
GraphQL subscription updates come in bursts. A pipeline with hundreds of jobs changes status many times a minute, and each trigger runs the subscription again for every subscriber. Today the only protection is a rate limit, and it drops the newest events, which carry the final state. In one week, GitLab.com dropped about 3.7 million subscription events this way, and users can see a pipeline that still shows "running" after it finished.
This MR adds the second policy from https://gitlab.com/gitlab-org/gitlab/-/work_items/630854: a subscription that sends the current state of one record can declare that its updates may be merged.
class StatusesUpdated < ::Subscriptions::BaseSubscription
coalesce
endThe default window is 500 milliseconds. A subscription can pass a longer one, such as coalesce window: 1.second.
On each websocket node, for each record:
- The first update runs at once and opens the window.
- Updates that arrive during the window are merged into one pending update.
- When the window ends, the pending update runs and opens the next window. When nothing is pending, the window closes.
Subscribers get at most one update for each record in each window, and they always get the last one. The update loads the record again, so it carries the current state.
In standard terms, this is a throttle with a leading and a trailing edge, per record, that keeps the newest update, like throttle in lodash or throttleTime with leading and trailing in RxJS. It is called a coalescer, not a throttle, because in this codebase "throttle" means the rate limiter, which drops events.
No subscription declares coalesce in this MR, so nothing changes yet. The next MR opts in ciPipelineStatusesUpdated. With merging in place, a later step can raise the producer rate limits to a safety value, so they almost never drop events.
Why on the websocket nodes
The work item first planned a Sidekiq debounce. It was rejected in https://gitlab.com/gitlab-org/gitlab/-/work_items/630854#note_4099485750 because each duplicate job costs WAL queries on the primary database, the debounced job bypasses the rate limit, and a duplicate that arrives while the job runs can be lost.
This MR merges in the two hooks that graphql-ruby's stream handler calls for each update, load_action_cable_message and execute_update, instead:
- Each process receives every message for its topics, from all producers, so it can merge without any coordination.
- GitLab has no broadcastable subscriptions, so each subscriber has its own fingerprint and its own stream handler. Each subscriber gets its own window.
- It adds no Sidekiq jobs, no Redis calls and no database queries. It removes executions and the
deliverbroadcasts that come with them. - A deferred update goes through
connection.worker_pool.async_invoke, the same path as an update from Redis. So it runs with the same ActionCable:workcallbacks: request store, instrumentation, load balancing and batch loader. The scheduler thread only hands work to the pool.
Details a reviewer should check
- Every update now takes the lazy path.
load_action_cable_messagereturns anUnloadedMessage, andexecute_updatedecides before it loads. This is true for every subscription, not only the ones that declarecoalesce. Behavior does not change for a subscription withoutcoalesce, but its path does:execute_updateparses the envelope once, and uses it both for the decision and for the fallback to the primary.- The loader then calls graphql-ruby's own
load_action_cable_messagedirectly, so a deferred update cannot reach the lazy override again. - The policy of each field is cached, so an update does not look up the schema field.
- The code depends on the two hooks that graphql-ruby exposes for backends, on the public
Event#fingerprint, and on graphql-ruby's@eventsmap, which the deferred run reads to find the current subscribers.
- Broadcastable subscriptions. A broadcastable subscription shares one fingerprint between subscribers. graphql-ruby calls
execute_updateonly for the first event of a fingerprint, so only that update can open a window. No subscription is broadcastable today. - Merge key.
Coalescer::Keyis the topic, the fingerprint and the global ID of the payload. A payload that is not one record, such as{ work_item_id:, action: }, is never merged.ciPipelineStatusesUpdatedhas a project-wide topic, but updates for different pipelines have different keys. - Concurrency. The coalescer keeps one
Burstfor each key in aConcurrent::Map. A flush drains the pending update, so the next window merges only updates that arrived after it. The cap check and the topic counter are separate steps, so the caps can go over by about the size of the worker pool. - WAL locations.
Envelope#mergekeeps the newer payload with the newest location of each database from both messages. Messages come from many processes, so the last one to arrive does not always carry the newest locations.Gitlab::Database::LoadBalancing::Lsncompares locations in Ruby, because the coalescer must not query the database. Sidekiq deduplication follows the same rule when it drops a duplicate job (DuplicateJob#update_latest_wal_location!), but it compares withpg_wal_lsn_diffon the primary. If any merged message has blank or unreadable locations, the result is blank, which reads from the primary. - Subscribers that leave. A deferred run looks up the current subscribers of the fingerprint with
fetch, because[]on graphql-ruby'sConcurrent::Mapcreates entries. If the subscriber left, nothing runs. - Memory. Each key with an open window stores one pending message, and a newer message is merged into it. The coalescer caps keys at 10,000 for each process and 1,000 for each topic. A key is one record for one subscriber, so a page that many people watch uses more keys of its topic. When a cap is full, the update runs at once, as today, and counts as
overflow. - Failures. No error in the coalescer loses an update:
- If a deferred update raises, the coalescer reports it and still opens the next window.
- If the scheduler cannot open a window, the coalescer forgets the key, so the key cannot stay deferred forever.
- If deciding to defer raises,
kept_for_later?reports the error and the update runs at once.
- Danger asks for a database review of lines with
deleteorupdate. They work on aConcurrent::Mapand on the coalescer'supdate:argument, not on a table.
Commits
-
The first commit adds the coalescer.
-
The next seven come from the review in !269582 (closed). Two of them fix bugs:
- A key could stop flushing when the scheduler raised.
- Old updates carried into the next window, so one blank WAL location pinned every later merged update of that key to the primary.
The other five move code out of
ActionCableWithLoadBalancingintoEnvelope,LsnandCoalescePolicy. -
Two commits cover branches that
rspec:coveragefound untested. -
The last commits come from the review threads on this MR:
- The coalescing decision moved from a
setup_streamoverride and a channel wrapper intoload_action_cable_messageandexecute_update. Lsnmoved toGitlab::Database::LoadBalancing, andActionCableWithLoadBalancingusesEnvelopefor every broadcast.- A spec shows that ending a window runs no queries.
- Each update is parsed once, and the policy of each field is cached.
- The coalescing decision moved from a
Metrics
gitlab_graphql_subscription_coalesced_events_total{subscription, outcome}:run,deferred,flushedandoverflow.action_cable_graphql_coalescer_keys: records with an open window, sampled by the existing ActionCable sampler.
Feature flag
graphql_subscription_coalescing (gitlab_com_derisk, off by default). The actor is the user who subscribes, which the query context already holds, so the check runs no query. The flag can be enabled for one user or a feature group, and a percentage rollout splits viewers evenly. Signed-out subscribers have no actor, so their updates are coalesced only when the flag is enabled for everyone. Rollout: #637380
Documentation
The "Limit and coalesce subscription updates" section of the GraphQL style guide explains both policies, rate_limit and coalesce. It says when to use each one and when not to, so other teams can opt in their subscriptions.
How to set up and validate locally
bin/rspec -r spec_helper spec/lib/gitlab/graphql/subscriptions/coalescer_spec.rb \
spec/lib/gitlab/graphql/subscriptions/action_cable_with_load_balancing_spec.rb \
spec/graphql/subscriptions/base_subscription_spec.rb \
spec/lib/gitlab/metrics/samplers/action_cable_sampler_spec.rbThe specs cover the window and its default, the merge of WAL locations, the key caps, subscribers that leave, the flag actor, and signed-out subscribers. spec/channels/graphql_channel_spec.rb and spec/requests/api/graphql/subscriptions also pass, so subscriptions that do not declare coalesce work as before.
Verified in GDK
With !266626 (merged) on top, which opts in ciPipelineStatusesUpdated. This check ran with a 2-second window, before the stream handler wrapper and the user actor:
- Open the commit pipelines tab of a commit with pipelines, so the browser subscribes to
ciPipelineStatusesUpdatedfor the project and the commit SHA. - In a Rails runner, call
GraphqlTriggers.ci_pipeline_statuses_updated(pipeline)10 times for one pipeline of that commit. - Count the
GraphqlChannel transmittinglines forciPipelineStatusesUpdatedinlog/development.log.
graphql_subscription_coalescing |
How the 10 triggers arrived | Results sent to the browser |
|---|---|---|
| Off | Within 12 ms | 10 |
| On | Within 12 ms | 2: the first at once, then one when the window ended |
| On | 0.5 s apart, over 4.6 s | 4: the first at once, then one at each of 3 window ends |
So the deferred update runs through the real ActionCable worker pool, and the window opens again while updates keep coming. The last result of each run carried the current state of the pipeline. The log had no There was an exception lines from the worker pool.
MR acceptance checklist
Evaluate this MR against the MR acceptance checklist.
References
- Part of https://gitlab.com/gitlab-org/gitlab/-/work_items/630854, step C1
- Design review that replaced the Sidekiq debounce with this coalescer: https://gitlab.com/gitlab-org/gitlab/-/work_items/630854#note_4099485750
- Builds on !264287 (merged), which declares rate limits on the subscription class
- Builds on !264263 (merged), which counts dropped events
- Production evidence of dropped events: https://gitlab.com/gitlab-org/gitlab/-/work_items/630854#note_4059560153
- Replaces the rejected Sidekiq debounce POC: !257713 (closed)
- Review with suggested changes, now part of this MR: !269582 (closed)
- Flag rollout: #637380
- Opt-ins: !266626 (merged) (both pipeline status subscriptions), !266649 and !266657