Limit Sentry error tracking GraphQL fields per operation
What does this MR do and why?
Four GraphQL fields read Sentry through ReactiveCaching, and each call with distinct arguments is its own cache entry (up to 5 MB in Rails.cache) and its own ExternalServiceReactiveCachingWorker job refreshed every minute: Project.sentryDetailedError, and detailedError, errorStackTrace and errors on SentryErrorCollection. The web JSON endpoints create one entry per request. GraphQL does not: a client can alias the same field many times in one operation, and each field costs only complexity 2, so one sentryErrors block with 124 aliased detailedError children fits the authenticated complexity budget of 250 and creates 124 entries and 124 jobs.
This MR counts those fields per GraphQL operation with the existing Gitlab::Graphql::Limit::FieldCallCount extension, which the work item error tracking widget already uses for its identifier, stackTrace and status fields:
| Field | Limit per operation |
|---|---|
Project.sentryDetailedError |
1 |
SentryErrorCollection.detailedError |
1 |
SentryErrorCollection.errorStackTrace |
1 |
SentryErrorCollection.errors |
2 |
FieldCallCountcounts per field per query fingerprint, across aliases and across parents, so aliasingprojector walking a project connection does not reset it. The alias over the limit gets aLimitError("detailedErrorfield can be requested only for 1 SentryErrorCollection(s) at a time.") and its resolver never runs, so no service call, cache key or job is created for it.- The extension is declared once per resolver (
SentryDetailedErrorResolver,SentryErrorStackTraceResolver,SentryErrorsResolver) through a smallResolvers::ErrorTracking::SentryFieldCallLimitconcern, the resolver-level form from the GraphQL styleguide, soProject.sentryDetailedErrorandSentryErrorCollection.detailedErrorare covered by one declaration. Because the counter is per field, an operation can still fetch one detail through each of those two fields, a bound of two rather than 124. - The extension gains an optional
enabled:lambda, called with the object the field is resolved on, so a field can roll the limit out behind a feature flag. The Sentry resolvers passerror_tracking_graphql_field_call_limit(default off) with the project as actor, so the rollout can start with a canary project like the sibling entry cap flag. Existing callers of the extension are unchanged, and the styleguide documents the option. - The four field descriptions state the limit, as the styleguide asks, and
doc/api/graphql/reference/_index.mdis regenerated. SentryErrorCollection.externalUrlonly reads the project setting and creates no entry, so it has no limit.- Raising the fields'
complexity:was considered as a complement and left out: the call count is precise and flag-gated, and a complexity bump would change the cost reported to every client immediately.
Survivable threshold
The limits are what the GitLab frontend and documented clients send. app/assets/javascripts/error_tracking/queries/details.query.graphql selects one detailedError per operation, the work item widget selects one stackTrace and one status, and the highest observed per-user rate is 4 requests per minute across operations that select these fields. errors allows two so a client can fetch a list and a page-ahead or a second filter in one operation. With the limits one operation creates at most 4 cache entries instead of 124, which puts GraphQL on par with the web JSON endpoints. A client needing more issues separate operations, which the per-user and per-project entry cap (!268728 (merged)) bounds per minute.
References
- Task: https://gitlab.com/gitlab-org/gitlab/-/work_items/638963 (confidential)
- Parent: https://gitlab.com/gitlab-org/gitlab/-/work_items/636812 (confidential)
- Sibling controls: !268260 (merged) (input validation), !268728 (merged) (entry cap)
- Rollout issue: https://gitlab.com/gitlab-org/gitlab/-/work_items/639014
Screenshots or screen recordings
Not applicable, backend-only change to the GraphQL API.
How to set up and validate locally
The Sentry settings page needs a reachable Sentry to save, so create the setting from the Rails console. The API URL only has to pass format validation.
-
Create a project, for example
root/et-test, then inbin/rails console:project = Project.find_by_full_path('root/et-test') project.create_error_tracking_setting!( enabled: true, integrated: false, api_url: 'https://sentry.io/api/0/projects/my-org/my-project/', token: 'dummy-token' ) Feature.enable(:error_tracking_graphql_field_call_limit, project) -
In GraphiQL at
http://127.0.0.1:3000/-/graphql-explorer, run an operation that aliasesdetailedErrortwice:query { project(fullPath: "root/et-test") { sentryErrors { a: detailedError(id: "gid://gitlab/Gitlab::ErrorTracking::DetailedError/1") { id } b: detailedError(id: "gid://gitlab/Gitlab::ErrorTracking::DetailedError/2") { id } } } }The response has
data.project.sentryErrors.a(nullwhile the cache warms, as before) and anerrorsentry:"detailedError" field can be requested only for 1 SentryErrorCollection(s) at a time. -
Confirm only the first alias reached the cache. In the console, one
issue_detailskey exists, for id1:Gitlab::Redis::Cache.with do |r| r.scan_each(match: 'cache:gitlab:error_tracking_project_error_tracking_setting:*:issue_details:*').to_a end -
Repeat with two aliased
projectfields each selecting onedetailedError; the second still returns the limit error. Three aliasederrors { nodes { id } }return the error for the third; two succeed. -
Feature.disable(:error_tracking_graphql_field_call_limit)and rerun step 2: both aliases resolve and two cache keys exist.
MR acceptance checklist
Evaluate this MR against the MR acceptance checklist. It helps you analyze changes to reduce risks in quality, performance, reliability, security, and maintainability.