Cap new error tracking Sentry cache entries per user and project
What does this MR do and why?
The Sentry-backed error tracking services (ErrorTracking::ListIssuesService, IssueDetailsService, IssueLatestEventService) read through ProjectErrorTrackingSetting#with_reactive_cache. Every distinct set of parameters is its own cache entry (up to 5 MB in Rails.cache, about 0.8 MB measured) and its own ExternalServiceReactiveCachingWorker job, refreshed every minute while the entry is read. !268260 (merged) rejects malformed parameters, but every valid search_term, issue_id or cursor is still a new entry, and nothing bounds how fast one user can create them.
This MR caps the rate at which new entries can be created. Both keys are defined with staged_rule, so they count in log mode until the error_tracking_sentry_cache_entry_limit feature flag (default off, project actor) turns on enforcement:
| Rate limit key | Scope | Threshold |
|---|---|---|
error_tracking_sentry_cache_entries_per_user |
user + project | 60 new entries per minute |
error_tracking_sentry_cache_entries_per_project |
project | 300 new entries per minute |
How it works:
ProjectErrorTrackingSetting#sentry_cache_alive?exposes the same alive-key checkwith_reactive_cacheuses to decide between "serve or wait" and "refresh and enqueue". The services call it first and skip the limiter when the entry exists, so only cache misses count. Polling the error list every second, reloading a details page, or many users reading the same list never touch the limiter.- The check runs on the Sentry branch of the three services, after input validation and before
with_reactive_cache. A throttled request creates no:alivekey, no data key and no job. Integrated error tracking is untouched. - The per-user limit is checked first and short-circuits, so a blocked user stops consuming the project-wide budget.
- With the flag off, the
_logrules count every would-be block without blocking. They show up in thegitlab_labkit_rate_limiter_rule_evaluations_totalmetric and in therate_limit_statefield of the request log line, which is how the thresholds get validated on GitLab.com before enforcement (seedoc/development/application_limits.md, "Roll out a key in log mode"). - A throttled request returns
429 Too Many Requestswith the standardApplicationRateLimiter.throttled_error_messagefrom the web JSON endpoints (list, details, stack trace). The GraphQL resolvers (sentryErrors.errors,sentryErrors.detailedError,sentryErrors.errorStackTrace) raiseResourceNotAvailablewith the same message instead of returningnullas they do for "not ready yet". - The work item error tracking widget (
Types::WorkItems::Widgets::ErrorTrackingType#status) treats a 429 like a 204, returningretry, since the entry is allowed again within the minute.
Survivable threshold
The frontend creates a new entry only when request parameters change: one per (issue_status, sort, search_term, cursor) on the list page (search submits on Enter, so no per-keystroke requests), and two per issue on the details page (issue_details and issue_latest_event). Polling and reloads of the same view are free. A very active user opening 20 issues and running 20 searches in one minute creates about 60 entries; ten people doing that at once on one project is about 600 per minute, far outside normal use. If a limit is hit, the user sees the existing "Failed to load" alert and polling resumes on the next tick.
With the cap, one account can create at most 60 entries per minute on a project, and a project cannot exceed 300 per minute across all accounts. Entries that are not re-read expire about 11 minutes after creation, so the steady state from one account is about 660 live entries, roughly 0.5 GB and 11 refresh jobs per second, against about 84 GB free in the Rails cache at weekday peak. Reads remain free, so an attacker who keeps re-reading everything they created can still grow the total by up to 300 entries per project per minute; bounding the total is the byte and key-set cap in the parent work item. The dedicated worker class and GraphQL per-request counting are separate tasks too.
References
- Task: https://gitlab.com/gitlab-org/gitlab/-/work_items/638561 (confidential)
- Parent: https://gitlab.com/gitlab-org/gitlab/-/work_items/636812 (confidential)
- Previous control: !268260 (merged)
- Rollout issue: https://gitlab.com/gitlab-org/gitlab/-/work_items/638808
Screenshots or screen recordings
Not applicable, backend-only change. A throttled request shows the existing "Failed to load" alert on the error tracking pages, as any other non-2xx response does.
How to set up and validate locally
The Sentry settings page needs a reachable Sentry to save, so create the setting from the Rails console. The API URL only has to pass format validation.
-
Create a project, for example
root/et-test, then inbin/rails console:project = Project.find_by_full_path('root/et-test') project.create_error_tracking_setting!( enabled: true, integrated: false, api_url: 'https://sentry.io/api/0/projects/my-org/my-project/', token: 'dummy-token' ) Feature.enable(:error_tracking_sentry_cache_entry_limit, project) -
Signed in as root, request 61 distinct search terms within a minute. The first 60 return
204(entry created, job enqueued), the 61st returns429with{"message":"This endpoint has been requested too many times. Try again later."}:for i in $(seq 1 61); do curl -s -o /dev/null -w "%{http_code}\n" -b "_gitlab_session=<cookie>" \ "http://127.0.0.1:3000/root/et-test/-/error_tracking.json?search_term=term-$i" done -
Repeat any of the first 60 URLs. It is a cache hit, so it returns the cached result (the Sentry
401with the dummy token) and never429, however often it is polled. -
Confirm the throttled request left nothing behind. In the console, exactly 60
list_issueskeys exist, none forterm-61:Gitlab::Redis::Cache.with do |r| r.scan_each(match: 'cache:gitlab:error_tracking_project_error_tracking_setting:*:list_issues:*').count endgdk tail rails-background-jobs | grep ExternalServiceReactiveCachingWorkershows no job forterm-61, and the request's line inlog/api_json.logorlog/production_json.logcarriesrate_limit_statewithapplimiter_error_tracking_sentry_cache_entries_per_user:limit_error_tracking_sentry_cache_entries_by_user_project:limit. -
The same applies to
/-/error_tracking/<id>/details.json,/-/error_tracking/<id>/stack_trace.jsonand the GraphQLsentryErrorsfields, which return a GraphQL error with the same message. -
Disable the flag and repeat step 2. All 61 requests return
204, and theirrate_limit_stateshows the_logrule with:logfor the 61st, which is the log-mode signal used before enforcing on GitLab.com.
MR acceptance checklist
Evaluate this MR against the MR acceptance checklist. It helps you analyze changes to reduce risks in quality, performance, reliability, security, and maintainability.