Count work item broadcasts dropped after a rate limit peek

What does this MR do and why?

Four rate limits guard GraphQL subscription triggers. When one of them blocks, the event is dropped and subscribers can show stale data, for example a pipeline that still shows "running" after it finished. We want to drive these drops toward zero, and we need a baseline first.

Labkit already counts the blocked calls of every limiter in gitlab_labkit_rate_limiter_rule_evaluations_total. It misses one case:

  • WorkItems::NamespaceChanges::BroadcastService first peeks at the namespace_work_item_changes_broadcast limit for every target namespace.
  • When all of them are over the limit, it returns early, before the expensive readability checks.
  • Labkit records a peek without an allow or block result, so these dropped events are not counted anywhere.

This MR counts them in a new gitlab_graphql_subscription_events_dropped_total counter, one event per skipped namespace. The labels are subscription (here namespace_work_item_changes) and reason (here rate_limited).

The behaviour does not change.

What was ruled out

Two earlier versions of this MR counted every dropped event, not only this case. Both had to turn blocked calls into events (one blocked pipeline call drops 4 events), either with a number written at each call site or by intercepting the triggers. That is more code than a baseline needs: the number of events per blocked call is fixed for each limiter, so it can be applied to the labkit data.

Counting per event comes naturally once the limit itself is checked per event. That is the next MR: !264287 (merged)

How to set up and validate locally

Run the specs:

bin/rspec spec/lib/gitlab/graphql/subscriptions/dropped_events_spec.rb spec/services/work_items/namespace_changes/broadcast_service_spec.rb

MR acceptance checklist

Evaluate this MR against the MR acceptance checklist.

References

Edited by Miguel Rincon

Merge request reports

Loading
Loading