Tune threshold for sidekiq exception alerts
We recently created a new alert for sidekiq exception count in #4651 (closed), but quickly noticed that we were seeing radically different volumes (~5x to ~15x higher) from one controller in particular, RepositoryUpdateMirrorWorker. After looking at https://sentry.gitlab.net/gitlab/gitlabcom/issues/217714/, it looks like the majority are for legitimate, albeit misconfigured jobs.
For an easy fix to make this alert more useful and cut down on the noise, we should implement a separate threshold for the RepositoryUpdateMirrorWorker. Based on https://dashboards.gitlab.net/d/9GOIu9Siz/sidekiq-stats?panelId=66&fullscreen&orgId=1&from=now-30d&to=now, setting it to 10000 for RepositoryUpdateMirrorWorker and 2000 for the others should be more appropriate
Activity
changed milestone to %2018-10-06 Completed Milestone
added unscheduled label
created branch
5201-tune-threshold-for-sidekiq-exception-alertsmentioned in merge request gitlab-com/runbooks!766 (merged)
mentioned in issue #5203 (closed)
closed via merge request gitlab-com/runbooks!766 (merged)
