Skip to content
Snippets Groups Projects

Tune thresholds for sidekiq exception counts

Merged Craig Barrett requested to merge 5201-tune-threshold-for-sidekiq-exception-alerts into master
1 file
+ 11
2
Compare changes
  • Side-by-side
  • Inline
@@ -2,12 +2,21 @@ groups:
@@ -2,12 +2,21 @@ groups:
- name: sidekiq-exception-count.rules
- name: sidekiq-exception-count.rules
rules:
rules:
- alert: SidekiqExceptionCountTooHigh
- alert: SidekiqExceptionCountTooHigh
expr: sum by (controller,environment) (gitlab_transaction_event_sidekiq_exception_total) > 5000
expr: sum by (controller,environment) (gitlab_transaction_event_sidekiq_exception_total{controller!="RepositoryUpdateMirrorWorker"}) > 2000
for: 1h
for: 1h
labels:
labels:
severity: warn
severity: warn
annotations:
annotations:
description: There have been over 5000 Sidekiq exceptions for {{ $labels.controller }} in {{ $labels.environment }} for the last hour.
description: There have been over 2000 Sidekiq exceptions for {{ $labels.controller }} in {{ $labels.environment }} for the last hour.
 
Check https://dashboards.gitlab.net/dashboard/db/sidekiq-stats?panelId=66&fullscreen&orgId=1&from=now-6h&to=now.
 
title: 'Large number of Sidekiq exceptions for {{ $labels.controller }} in {{ $labels.environment }}: {{$value}}'
 
- alert: RepositoryMirrorUpdateWorkerExceptionCountTooHigh
 
expr: sum by (controller,environment) (gitlab_transaction_event_sidekiq_exception_total{controller="RepositoryUpdateMirrorWorker"}) > 10000
 
for: 1h
 
labels:
 
severity: warn
 
annotations:
 
description: There have been over 10000 Sidekiq exceptions for {{ $labels.controller }} in {{ $labels.environment }} for the last hour.
Check https://dashboards.gitlab.net/dashboard/db/sidekiq-stats?panelId=66&fullscreen&orgId=1&from=now-6h&to=now.
Check https://dashboards.gitlab.net/dashboard/db/sidekiq-stats?panelId=66&fullscreen&orgId=1&from=now-6h&to=now.
title: 'Large number of Sidekiq exceptions for {{ $labels.controller }} in {{ $labels.environment }}: {{$value}}'
title: 'Large number of Sidekiq exceptions for {{ $labels.controller }} in {{ $labels.environment }}: {{$value}}'
runbook: troubleshooting/sidekiq_exception_count.md
runbook: troubleshooting/sidekiq_exception_count.md
Loading