Adjust SLO alerting threshold for the Grafana service
Summary
The alert for the public Grafana is too sensitive adding to on-call toil. Before we can adjust it, we need to split out the Grafana service out of the monitoring service: https://gitlab.com/gitlab-com/gl-infra/infrastructure/-/issues/14335 . For more details (including alternatives) see: https://gitlab.com/gitlab-com/gl-infra/production/-/issues/5692#note_700578499
Related Incident(s)
Originating issue(s): https://gitlab.com/gitlab-com/gl-infra/production/-/issues/5692#
Desired Outcome/Acceptance criteria
EOC doesn't get paged after a low number (single) of failed http requests.
Associated Services
Corrective Action Issue Checklist
-
link the incident(s) this corrective action arose out of -
give context for what problem this corrective action is trying to prevent from re-occurring -
assign a severity label (this is the highest sev of related incidents, defaults to 'severity::4') -
assign a priority (this will default to 'priority::4')