Declare GraphQL subscription rate limits on the subscription
What does this MR do and why?
Four GraphQL subscriptions have a rate limit, so that a burst of changes in one project cannot flood ActionCable. Until now, every one of those limits was written by hand at its call site, each in a different place and a different form. Nothing on the subscription itself said that it was limited, so a new subscription, or a new caller of an existing one, got no limit unless someone remembered to add the check.
This MR moves the limit onto the subscription. A subscription now declares its limit once:
class JobProcessed < Subscriptions::BaseSubscription
rate_limit :ci_job_processed_subscription, scope: ->(args, _job) { { project: args[:project_id].model_id } }The declaration is stored as a small policy object, and a subclass inherits it. Every subscription event already goes through one method, Gitlab::Graphql::Subscriptions::ActionCableWithLoadBalancing#trigger. That method now checks the declared limit for each event. When the limit blocks an event, the event is dropped and counted in gitlab_graphql_subscription_events_dropped_total, which !264263 (merged) added. The four hand-written checks go away.
This gives each subscription one place for its policy. The next MR in the stack, !266624 (merged), uses the same place for a coalescing window, so that bursts can be merged instead of dropped.
What changes
This is mostly a restructuring, but it is not a pure refactor. In steady state, the same events are sent and dropped. These things change:
- The limits count events, not calls. One pipeline transition triggers 4
ciPipelineStatusesUpdatedtopics, and one job transition triggers 2ciJobProcessedtopics. Their limits go from 50 calls to 200 and 100 events per minute, which keeps the budget at 50 transitions per minute per project. At the edge of a window, a transition can now be cut between its topics: some topics go out and the rest are dropped. Before, a transition was all or nothing. - The two CI rules get new names,
limit_ci_pipeline_status_events_by_projectandlimit_ci_job_processed_events_by_project. The rule name is part of the labkit Redis key, so old and new pods do not share a counter during the deploy:- Each version applies its own limit, so no pod is stricter than before. At worst, a project gets twice its budget while both versions run.
- The budgets restart once at deploy.
- The rule name is also the
rulelabel ofgitlab_labkit_rate_limiter_rule_evaluations_total. Per-event counts appear under the new names, so no series mixes calls and events.
- The work item limit gets new Redis keys, so its budgets restart once at deploy. Which namespaces share a budget does not change. See the details below. This is a separate commit.
- Any new caller of these four fields is limited automatically. Today each field has one caller, so nothing changes now.
- Drops of all four subscriptions are now counted in
gitlab_graphql_subscription_events_dropped_total. - The CI limits make more calls to the rate-limiting Redis. Each topic is counted, so a pipeline transition makes 4 calls instead of 1, and a job transition makes 2 instead of 1. The job limit only runs while the
ci_job_created_subscriptionflag is on.
There is no change to the GraphQL API.
Where the limits were before
| Limiter key | Where the check was | Form |
|---|---|---|
ci_pipeline_statuses_subscription |
Ci::Pipeline, before it called the trigger |
one check for 4 triggers |
ci_job_processed_subscription |
Ci::Build, before it called the trigger |
one check for 2 triggers |
work_item_saved_view_broadcast |
inside GraphqlTriggers.work_item_saved_view_updated |
one check, one trigger |
namespace_work_item_changes_broadcast |
WorkItems::NamespaceChanges::BroadcastService, in its loop |
one check per namespace |
Budget per limit
| Limiter key | Topics per transition | Old limit (calls) | New limit (events) |
|---|---|---|---|
ci_pipeline_statuses_subscription |
4 | 50 | 200 |
ci_job_processed_subscription |
2 | 50 | 100 |
work_item_saved_view_broadcast |
1 | 30 | 30 |
namespace_work_item_changes_broadcast |
1 per namespace | 5 per 15 s | 5 per 15 s |
Every transition triggers all of its topics in a row, so a project still gets 50 transitions per minute, and the 51st is dropped as before.
How the scopes are built
Where the topic arguments carry a global ID, the scope is built from it, without a database query and without the object:
ciJobProcessed:{ project: args[:project_id].model_id }workItemSavedViewUpdated:{ saved_view: args[:saved_view_id].model_id }namespaceWorkItemChanges:{ namespace: args[:namespace_id].model_id }. Its object is a plain hash with no namespace, and one change triggers once per namespace in the hierarchy.
A subscription framework must also handle objects that are gone, for example after a delete, so the scope should not depend on the object when it can avoid it. The limiter turns a record into its id and a primitive into its string form, so these Redis keys stay the same.
ciPipelineStatusesUpdated is the exception and keeps its scope from the object, { project: pipeline.project }. Two of its four topics only carry projectFullPath, which makes UIs easier to build: the new merge request page knows the source project's path, not its global ID (!266580). All four topics share one budget, so they must share one scope.
Why the work item limit changes its Redis keys
namespace_work_item_changes_broadcast had two characteristics, group and namespace, and filled one of them by class. The split came from the move to hash scopes in 902c553b, which kept every Redis key of the old class-based scopes. Groups and other namespaces share one table, so the namespace id alone is unique.
The rule now has the single characteristic namespace, and the subscription scopes like the others. The limit does not change, but the key names do.
BroadcastService still peeks at the limit before its readability checks. That shortcut skips expensive queries when nothing can be delivered. It peeks through the policy of NamespaceWorkItemChanges, so it reads the same counter that the trigger counts against. A general way to decide before the work is an open question in https://gitlab.com/gitlab-org/gitlab/-/work_items/630854.
How the limit is stored
rate_limit stores a Gitlab::Graphql::Subscriptions::RateLimitPolicy in the class attribute rate_limit_policy. The policy has two methods:
throttled?(args, object)counts the event. The trigger calls it.exhausted?(args, object)only reads the budget.BroadcastServicecalls it.
A class attribute is inherited, so a subclass of a limited subscription is limited too. A subclass that declares its own limit does not change the limit of its parent.
Cost for subscriptions without a limit
A subscription without rate_limit costs one field lookup per trigger and nothing else. For an unknown field, graphql-ruby still raises its own InvalidTriggerError, and a spec covers that.
How to set up and validate locally
Run the specs:
bin/rspec spec/graphql/subscriptions/base_subscription_spec.rb spec/lib/gitlab/graphql/subscriptions/rate_limit_policy_spec.rb spec/lib/gitlab/graphql/subscriptions/action_cable_with_load_balancing_spec.rb
bin/rspec spec/graphql/graphql_triggers_spec.rb spec/services/work_items/namespace_changes/broadcast_service_spec.rb
bin/rspec spec/models/ci/build_spec.rb spec/models/ci/pipeline_spec.rb -e "trigger_status_change_subscriptions" -e "job creation subscription trigger"
bin/rspec spec/requests/api/graphql/subscriptions spec/channels/graphql_channel_spec.rbMR acceptance checklist
Evaluate this MR against the MR acceptance checklist.
References
- Part of https://gitlab.com/gitlab-org/gitlab/-/work_items/630854 (the decided design for one place for rate limiting and coalescing)
- Uses the dropped-events counter from !264263 (merged)
- Next in the stack, the coalescer that reads its window from the subscription class: !266624 (merged)
- Uses
projectFullPathof ciPipelineStatusesUpdated, the reason its scope comes from the object: !266580 - Closed, removal of
projectFullPaththat this MR no longer depends on: !267017 (closed) - Source of the group/namespace split: 902c553b
- Review note on the shared counter during the deploy, the reason for the new CI rule names: !264287 (comment 4170041385)
- Review note on the unused
ciJobProcessedshared examples in the build spec: !264287 (comment 4176032470) - Review note that suggested the policy object: !264287 (comment 4170116069)