Run error tracking cache refreshes on a dedicated worker
What does this MR do and why?
Sentry error tracking cache entries (ProjectErrorTrackingSetting#list_sentry_issues, #issue_details, #issue_latest_event) are refreshed once a minute per live entry by ExternalServiceReactiveCachingWorker. That class is the single ReactiveCaching worker for every model declared with reactive_cache_work_type = :external_dependency: environments, clusters, Prometheus, SSH host keys, integrations and Jira issue lookups. It has no static concurrency_limit, so all of those share the GitLab.com default limit, and a burst of error tracking refreshes defers Jira, environment and cluster jobs along with its own. Its feature_category is :not_owned, so the load is not even attributed to error tracking.
This MR moves error tracking refreshes to a worker of their own:
Observability::ErrorTrackingReactiveCachingWorker < ExternalServiceReactiveCachingWorker, following howReactiveCaching::LowUrgencyWorkersubclassesReactiveCachingWorker. It inheritsReactiveCacheableWorker#perform,worker_has_external_dependencies!,worker_resource_boundary :cpu,data_consistency :stickyandretry: 3, and addsfeature_category :observabilityand aconcurrency_limit.- A new
:error_trackingentry inReactiveCaching::WORK_TYPE, used byProjectErrorTrackingSetting.ReactiveCaching#worker_classand#background_update_workerboth read the work type, so the first refresh and every per-minute re-enqueue move together. The other:external_dependencymodels are unchanged. app/workers/all_queues.ymlandconfig/sidekiq_queues.ymlregenerated with the rake tasks. On GitLab.com the worker is routed by its attributes, so it lands on the same shard as before.
The worker lives under Observability:: rather than ErrorTracking:: because ErrorTracking is not a registered bounded context (config/bounded_contexts.yml) and the Gitlab/BoundedContexts cop rejects new classes outside one; Observability owns the observability feature category.
Feature flag
error_tracking_dedicated_reactive_caching_worker (gitlab_com_derisk, default off, checked against :instance because the first enqueue happens in web and the per-minute re-enqueue in Sidekiq, and both must agree) gates the work type. With the flag off, ProjectErrorTrackingSetting.reactive_cache_work_type is :external_dependency, exactly today's behaviour. With the flag on, new entries and re-enqueues go to the dedicated worker, where the static concurrency limit applies. Rollback is the flag: jobs already on the new class finish there and new ones return to the shared worker.
Gating the enqueue, not only the limit, follows Adding new workers: GitLab.com has no Sidekiq canary, so a new worker class scheduled from web before the Sidekiq deploy completes would wait for hours. The flag is enabled after the class is deployed everywhere.
Survivable threshold
Each refresh job is one Sentry request, about a second, CPU-bound while parsing up to 5 MB of JSON. One live entry costs one job per minute, so a concurrency limit of N sustains roughly 60 × N live entries before jobs defer.
- Starting value:
CONCURRENCY_LIMIT = 20, about 1,200 live entries refreshed per minute. With the entry cap from !268728 (merged) (60 new entries per user per project per minute, 300 per project), one account cannot reach that; several projects or many accounts can, and the overflow then defers on this class only. - To confirm before enabling:
sidekiq_jobs_completion_seconds_count{worker="ExternalServiceReactiveCachingWorker"}filtered byjson.meta.related_class: ErrorTracking::ProjectErrorTrackingSettinggives today's error tracking refresh rate on GitLab.com; after this MR deploys the new worker name gives it directly. The limit should sit a few times above the peak. - When the limit is hit, refreshes are deferred, not dropped.
ReactiveCachingonly re-enqueues a refresh after the previous one finishes andexclusively_update_reactive_cache!holds a lease per entry, so the deferred list holds at most one job per live entry and shrinks as entries expire. Users see data a few minutes stale, not errors.sidekiq_concurrency_limit_deferred_jobs_totalandsidekiq_concurrency_limit_queue_jobsshow it.
The value is a static number rather than a fleet-relative max_concurrency_limit_percentage on purpose: the point is a cap well below the shard's capacity, sized from the measured error tracking refresh rate. Self-managed instances keep the default fleet-relative limit while the flag is off. If a per-instance value is wanted after the flag is removed, the number can move to an application setting, as import_jobs_concurrency_limit does for BulkImports::PipelineWorker.
Out of scope: per top-level group scoping with concurrency_limit_scope(:root_namespace) (possible follow-up on this worker), Redis byte and key-set caps, making the limit an application setting, and splitting the remaining :external_dependency models.
References
- Task: https://gitlab.com/gitlab-org/gitlab/-/work_items/638960 (confidential)
- Parent: https://gitlab.com/gitlab-org/gitlab/-/work_items/636812 (confidential)
- Rollout issue: https://gitlab.com/gitlab-org/gitlab/-/work_items/639013
- Previous controls: !268260 (merged), !268728 (merged)
Screenshots or screen recordings
Not applicable, backend-only change with no user-visible behaviour change.
How to set up and validate locally
The Sentry settings page needs a reachable Sentry to save, so create the setting from the Rails console. The API URL only has to pass format validation.
-
Create a project, for example
root/et-test, then inbin/rails console:project = Project.find_by_full_path('root/et-test') project.create_error_tracking_setting!( enabled: true, integrated: false, api_url: 'https://sentry.io/api/0/projects/my-org/my-project/', token: 'dummy-token' ) Feature.enable(:error_tracking_dedicated_reactive_caching_worker) -
Signed in, open
http://127.0.0.1:3000/root/et-test/-/error_tracking.json. It returns204and enqueues the first refresh. -
gdk tail rails-background-jobs | grep ReactiveCachingshowsObservability::ErrorTrackingReactiveCachingWorkerjobs with"meta.related_class":"ErrorTracking::ProjectErrorTrackingSetting"and"feature_category":"observability", re-enqueued every minute while the page is polled. NoExternalServiceReactiveCachingWorkerjob appears for it. -
Open an environments or clusters page in another project and confirm those still run on
ExternalServiceReactiveCachingWorker. -
Disable the flag with
Feature.disable(:error_tracking_dedicated_reactive_caching_worker)and reload the page: new jobs appear onExternalServiceReactiveCachingWorkeragain, as before this MR.
MR acceptance checklist
Evaluate this MR against the MR acceptance checklist. It helps you analyze changes to reduce risks in quality, performance, reliability, security, and maintainability.