Draft: [POC] Add a debounce for GraphQL subscription triggers
What does this MR do and why?
POC MR:
Adds a trailing-edge debounce for GraphQL subscription triggers, and uses it for the first consumer: the push to the pipeline subscription when a job fails while the pipeline keeps running.
A trigger re-runs the subscription for every subscriber. When many events hit one object in a short time, every event costs a full fan-out, an authorization check per subscriber, and any connection fields the client selected. The frontend already answers this with lodash debounce. The backend had no equivalent. The only guards were fixed-window rate limits, which drop the last event of a burst and leave the client stale until its next poll.
The debounce reuses the Sidekiq deduplication middleware that is already in production:
- A trigger opts in with an annotation above its method, and picks its own window:
debounced :ci_pipeline_status_updated, window: 2.seconds. Triggers that do not opt in cannot be debounced. - Callers that produce bursts use
GraphqlTriggers.debounce(trigger, object), which schedulesGitlab::Graphql::DebouncedTriggerWorker.perform_in(window, ...). Callers that need an immediate push keep calling the trigger directly. - The worker declares
deduplicate :until_executed, including_scheduled: true, if_deduplicated: :reschedule_once. A burst on one object collapses into one job. An event that lands while the job runs reschedules it once, so the trailing event is not lost. - The job reloads the object and fires the declared trigger with the object's current state.
The 2 second window is a first guess taken from the existing 2 second debounce in ProgressDeliveryWorker and the 3 second frontend batching on the pipelines list. It is not derived from production burst shapes. Reviewers should challenge it. In practice the delay is the window plus Sidekiq scheduler poll latency, about 2 to 7 seconds.
This is the same pattern as ee/app/workers/ai/messaging/progress_delivery_worker.rb and app/workers/flush_counter_increments_worker.rb. No new gem is needed. graphql-ruby has no hook to merge updates, and GraphQL Pro cannot ship in the product, so a change at the transport layer was ruled out.
The consumer commit carries over the change from MR 257495 and swaps the direct trigger for the debounced call. The ci_pipeline_failed_jobs_subscription_trigger flag stays so the rollout can watch broadcast volume. If this MR is preferred, MR 257495 can be closed.
Not in scope: switching the existing project-wide ci_pipeline_statuses_subscription rate limit to the debounce. That is a follow-up listed in issue 630854.
Screenshots or screen recordings
Not a UI change.
How to set up and validate locally
-
Enable the flag in a Rails console:
Feature.enable(:ci_pipeline_failed_jobs_subscription_trigger) -
Open a running pipeline page in the browser.
-
Fail many jobs of that pipeline in a burst, for example with a Rails runner that calls
drop!(:script_failure)on each running build. -
Check
log/sidekiq_client.logandlog/sidekiq.logforGitlab::Graphql::DebouncedTriggerWorker. Expect onestart/donepair andN - 1deduplicatedentries. -
The failed jobs widget on the pipeline page updates once, about two seconds after the first failure plus Sidekiq scheduler latency.
Verified in GDK
Two bursts against gitlab-org/gitlab-test, with one extra job left running so the pipeline never transitions on its own:
| Jobs failed | Burst length | Jobs pushed | Deduplicated | Worker runs | Trigger fired after burst end |
|---|---|---|---|---|---|
| 30 | 4.7 s | 1 | 29 | 1 | yes, 5 s later |
| 200 | 8.1 s | 1 | 199 | 1 | yes, 8.5 s later |
In the 200-job run the first push at 05:33:22.8 scheduled the job for 05:33:24.8. It ran once at 05:33:35.3, after the burst ended at 05:33:26.8, and the pipeline had 200 failed jobs at that time. So subscribers received the final state in one payload.
Specs added:
spec/workers/gitlab/graphql/debounced_trigger_worker_spec.rb: dedup options, reschedulable after dequeue, undeclared trigger refused, missing record, idempotency.spec/graphql/graphql_triggers_spec.rb:.debounceuses the declared window, refuses undeclared triggers, and three calls enqueue one job.spec/models/commit_status_spec.rb: five failures in one pipeline enqueue one debounced push.
MR acceptance checklist
Evaluate this MR against the MR acceptance checklist. It helps you analyze changes to reduce risks in quality, performance, reliability, security, and maintainability.
References
- Implements https://gitlab.com/gitlab-org/gitlab/-/work_items/630854
- Carries over and supersedes the trigger from !257495 (closed)
- Discussion that raised the debounce question: !257495 (comment 3902000412)
- Per-job stage subscription rollout closed over cardinality: #581569 (closed)
- Rollout of the failed-jobs trigger: #630685