Draft: [POC] Add a debounce for GraphQL subscription triggers

What does this MR do and why?

POC MR:

Adds a trailing-edge debounce for GraphQL subscription triggers, and uses it for the first consumer: the push to the pipeline subscription when a job fails while the pipeline keeps running.

A trigger re-runs the subscription for every subscriber. When many events hit one object in a short time, every event costs a full fan-out, an authorization check per subscriber, and any connection fields the client selected. The frontend already answers this with lodash debounce. The backend had no equivalent. The only guards were fixed-window rate limits, which drop the last event of a burst and leave the client stale until its next poll.

The debounce reuses the Sidekiq deduplication middleware that is already in production:

  • A trigger opts in with an annotation above its method, and picks its own window: debounced :ci_pipeline_status_updated, window: 2.seconds. Triggers that do not opt in cannot be debounced.
  • Callers that produce bursts use GraphqlTriggers.debounce(trigger, object), which schedules Gitlab::Graphql::DebouncedTriggerWorker.perform_in(window, ...). Callers that need an immediate push keep calling the trigger directly.
  • The worker declares deduplicate :until_executed, including_scheduled: true, if_deduplicated: :reschedule_once. A burst on one object collapses into one job. An event that lands while the job runs reschedules it once, so the trailing event is not lost.
  • The job reloads the object and fires the declared trigger with the object's current state.

The 2 second window is a first guess taken from the existing 2 second debounce in ProgressDeliveryWorker and the 3 second frontend batching on the pipelines list. It is not derived from production burst shapes. Reviewers should challenge it. In practice the delay is the window plus Sidekiq scheduler poll latency, about 2 to 7 seconds.

This is the same pattern as ee/app/workers/ai/messaging/progress_delivery_worker.rb and app/workers/flush_counter_increments_worker.rb. No new gem is needed. graphql-ruby has no hook to merge updates, and GraphQL Pro cannot ship in the product, so a change at the transport layer was ruled out.

The consumer commit carries over the change from MR 257495 and swaps the direct trigger for the debounced call. The ci_pipeline_failed_jobs_subscription_trigger flag stays so the rollout can watch broadcast volume. If this MR is preferred, MR 257495 can be closed.

Not in scope: switching the existing project-wide ci_pipeline_statuses_subscription rate limit to the debounce. That is a follow-up listed in issue 630854.

Screenshots or screen recordings

Not a UI change.

How to set up and validate locally

  1. Enable the flag in a Rails console:

    Feature.enable(:ci_pipeline_failed_jobs_subscription_trigger)
  2. Open a running pipeline page in the browser.

  3. Fail many jobs of that pipeline in a burst, for example with a Rails runner that calls drop!(:script_failure) on each running build.

  4. Check log/sidekiq_client.log and log/sidekiq.log for Gitlab::Graphql::DebouncedTriggerWorker. Expect one start/done pair and N - 1 deduplicated entries.

  5. The failed jobs widget on the pipeline page updates once, about two seconds after the first failure plus Sidekiq scheduler latency.

Verified in GDK

Two bursts against gitlab-org/gitlab-test, with one extra job left running so the pipeline never transitions on its own:

Jobs failed Burst length Jobs pushed Deduplicated Worker runs Trigger fired after burst end
30 4.7 s 1 29 1 yes, 5 s later
200 8.1 s 1 199 1 yes, 8.5 s later

In the 200-job run the first push at 05:33:22.8 scheduled the job for 05:33:24.8. It ran once at 05:33:35.3, after the burst ended at 05:33:26.8, and the pipeline had 200 failed jobs at that time. So subscribers received the final state in one payload.

Specs added:

  • spec/workers/gitlab/graphql/debounced_trigger_worker_spec.rb: dedup options, reschedulable after dequeue, undeclared trigger refused, missing record, idempotency.
  • spec/graphql/graphql_triggers_spec.rb: .debounce uses the declared window, refuses undeclared triggers, and three calls enqueue one job.
  • spec/models/commit_status_spec.rb: five failures in one pipeline enqueue one debounced push.

MR acceptance checklist

Evaluate this MR against the MR acceptance checklist. It helps you analyze changes to reduce risks in quality, performance, reliability, security, and maintainability.

References

Edited by Miguel Rincon

Merge request reports

Loading
Loading