Cap new error tracking Sentry cache entries per user and project

What does this MR do and why?

The Sentry-backed error tracking services (ErrorTracking::ListIssuesService, IssueDetailsService, IssueLatestEventService) read through ProjectErrorTrackingSetting#with_reactive_cache. Every distinct set of parameters is its own cache entry (up to 5 MB in Rails.cache, about 0.8 MB measured) and its own ExternalServiceReactiveCachingWorker job, refreshed every minute while the entry is read. !268260 (merged) rejects malformed parameters, but every valid search_term, issue_id or cursor is still a new entry, and nothing bounds how fast one user can create them.

This MR caps the rate at which new entries can be created. Both keys are defined with staged_rule, so they count in log mode until the error_tracking_sentry_cache_entry_limit feature flag (default off, project actor) turns on enforcement:

Rate limit key Scope Threshold
error_tracking_sentry_cache_entries_per_user user + project 60 new entries per minute
error_tracking_sentry_cache_entries_per_project project 300 new entries per minute

How it works:

  • ProjectErrorTrackingSetting#sentry_cache_alive? exposes the same alive-key check with_reactive_cache uses to decide between "serve or wait" and "refresh and enqueue". The services call it first and skip the limiter when the entry exists, so only cache misses count. Polling the error list every second, reloading a details page, or many users reading the same list never touch the limiter.
  • The check runs on the Sentry branch of the three services, after input validation and before with_reactive_cache. A throttled request creates no :alive key, no data key and no job. Integrated error tracking is untouched.
  • The per-user limit is checked first and short-circuits, so a blocked user stops consuming the project-wide budget.
  • With the flag off, the _log rules count every would-be block without blocking. They show up in the gitlab_labkit_rate_limiter_rule_evaluations_total metric and in the rate_limit_state field of the request log line, which is how the thresholds get validated on GitLab.com before enforcement (see doc/development/application_limits.md, "Roll out a key in log mode").
  • A throttled request returns 429 Too Many Requests with the standard ApplicationRateLimiter.throttled_error_message from the web JSON endpoints (list, details, stack trace). The GraphQL resolvers (sentryErrors.errors, sentryErrors.detailedError, sentryErrors.errorStackTrace) raise ResourceNotAvailable with the same message instead of returning null as they do for "not ready yet".
  • The work item error tracking widget (Types::WorkItems::Widgets::ErrorTrackingType#status) treats a 429 like a 204, returning retry, since the entry is allowed again within the minute.

Survivable threshold

The frontend creates a new entry only when request parameters change: one per (issue_status, sort, search_term, cursor) on the list page (search submits on Enter, so no per-keystroke requests), and two per issue on the details page (issue_details and issue_latest_event). Polling and reloads of the same view are free. A very active user opening 20 issues and running 20 searches in one minute creates about 60 entries; ten people doing that at once on one project is about 600 per minute, far outside normal use. If a limit is hit, the user sees the existing "Failed to load" alert and polling resumes on the next tick.

With the cap, one account can create at most 60 entries per minute on a project, and a project cannot exceed 300 per minute across all accounts. Entries that are not re-read expire about 11 minutes after creation, so the steady state from one account is about 660 live entries, roughly 0.5 GB and 11 refresh jobs per second, against about 84 GB free in the Rails cache at weekday peak. Reads remain free, so an attacker who keeps re-reading everything they created can still grow the total by up to 300 entries per project per minute; bounding the total is the byte and key-set cap in the parent work item. The dedicated worker class and GraphQL per-request counting are separate tasks too.

References

Screenshots or screen recordings

Not applicable, backend-only change. A throttled request shows the existing "Failed to load" alert on the error tracking pages, as any other non-2xx response does.

How to set up and validate locally

The Sentry settings page needs a reachable Sentry to save, so create the setting from the Rails console. The API URL only has to pass format validation.

  1. Create a project, for example root/et-test, then in bin/rails console:

    project = Project.find_by_full_path('root/et-test')
    project.create_error_tracking_setting!(
      enabled: true,
      integrated: false,
      api_url: 'https://sentry.io/api/0/projects/my-org/my-project/',
      token: 'dummy-token'
    )
    Feature.enable(:error_tracking_sentry_cache_entry_limit, project)
  2. Signed in as root, request 61 distinct search terms within a minute. The first 60 return 204 (entry created, job enqueued), the 61st returns 429 with {"message":"This endpoint has been requested too many times. Try again later."}:

    for i in $(seq 1 61); do
      curl -s -o /dev/null -w "%{http_code}\n" -b "_gitlab_session=<cookie>" \
        "http://127.0.0.1:3000/root/et-test/-/error_tracking.json?search_term=term-$i"
    done
  3. Repeat any of the first 60 URLs. It is a cache hit, so it returns the cached result (the Sentry 401 with the dummy token) and never 429, however often it is polled.

  4. Confirm the throttled request left nothing behind. In the console, exactly 60 list_issues keys exist, none for term-61:

    Gitlab::Redis::Cache.with do |r|
      r.scan_each(match: 'cache:gitlab:error_tracking_project_error_tracking_setting:*:list_issues:*').count
    end

    gdk tail rails-background-jobs | grep ExternalServiceReactiveCachingWorker shows no job for term-61, and the request's line in log/api_json.log or log/production_json.log carries rate_limit_state with applimiter_error_tracking_sentry_cache_entries_per_user:limit_error_tracking_sentry_cache_entries_by_user_project:limit.

  5. The same applies to /-/error_tracking/<id>/details.json, /-/error_tracking/<id>/stack_trace.json and the GraphQL sentryErrors fields, which return a GraphQL error with the same message.

  6. Disable the flag and repeat step 2. All 61 requests return 204, and their rate_limit_state shows the _log rule with :log for the 61st, which is the log-mode signal used before enforcing on GitLab.com.

MR acceptance checklist

Evaluate this MR against the MR acceptance checklist. It helps you analyze changes to reduce risks in quality, performance, reliability, security, and maintainability.

Edited by Nikhil Erkimutt

Merge request reports

Loading
Loading