Create 3 SLIs relating to 500 errors from recent production incidents
Background
In production-engineering#25401 (closed) we discuss how some 500 errors were precursors to incidents. However, the 500 errors were ignored.
As part of the discussion, we confirm that we should create two specific SLIs in reaction to the incidents that occurred.
Todo
Create two new SLIs. As part of creating them, tell the stage groups who own them what we have done and why.
1. An SLI for SAML Login.
@andrewn proposed the following: Since the GitLab application already emits the metric, a definition of the SLI might look like:
saml_logins: {
serviceAggregation: false,
userImpacting: true,
featureCategory: 'user_management',
description: |||
Measures SAML Logins as the number of successful SAML Logins vs the number of initiated attempts.
An alert on this SLI may indicate that SAML users are unable to log into GitLab.
|||,
requestRate: rateMetric(
counter='gitlab_omniauth_login_total',
selector={ status: "initiated", omniauth_provider: "group_saml" },
),
errorRate: rateMetric(
counter='gitlab_omniauth_login_total',
selector={ status: "failed", omniauth_provider: "group_saml" },
),
significantLabels: [],
toolingLinks: [],
}
2. An SLI for the Dependency Proxy.
An appropriate SLI could probably be defined by filtering down the GitLab Rails SLIs to only include the {endpoint_id=~"Groups::DependencyProxyForContainersController.*"}
. There might be another way to do this.
3. An SLI for project and group management
This could be an SLI that narrows down the rails SLI to the specific endpoints for project and group management using endpoint_id=~"PUT /api/:version/groups/:id|POST /api/:version/groups"
as a selector. We might want to do the same for the web
service with the related controller endpoints, which would also have failed during this incident, but they were not called out.