DRI @reprazent ## Background In &437, we introduced Error Budgets to Stage Groups and followed this with &468 where we made the Error Budget information easier to consume. Now that Error Budgets are part of the prioritization process to encourage the [SaaS-first mindset](https://about.gitlab.com/direction/#saas-first), the next step is to enable Stage Groups to set their to define and adjust Service Level Indicators (SLIs). We currently have several SLIs feeding into the error budgets, most of these are defined in the service catalog. Though this currently only works for SLIs on services that directly correlate to a feature category. The metrics keeping taps on the `web`, `api`, `git` and `sidekiq` fleets are currently different from the metrics that are feeding in to the error budgets for stage groups. Though they result in mostly the same results, this does make it possible for SLIs to cause a differences in error budget spent. For example the current SLI for request duration (puma apdex) has a threshold set to 1 second for the stage group error budgets. This is currently a threshold set based on the SLI we had for request durations in the metrics catalog. But the version in the metrics catalog is based on a histogram with a wider range of buckets which we cannot add with the granularity required for stage group error budgets. The 1s threshold for request duration is a good start, for an overall view of all requests. But it is not possible to actually enforce this for all indpoints: Some busy endpoints should be faster than 1s, while we could allow others less busy ones to be slower. While we want this to be largely self-service, we need to retain control over the thresholds that are set to prevent misuse of resources. Error budgets were designed to help teams understand their spend. If teams excessively increase their thresholds, the Error Budgets will lose their value. So part of this project needs to be setting up a process where teams submit the proposed changes for review before these are merged. We need to ensure that the SLIs requested are not out of the reach of the infrastructure that supports the requests. To be able to move forward and allow stage groups to define their own indicators and thresholds for them inside the Rails codebase and use those for both service monitoring and the stage group error budgets. To avoid mismatches in the future. ## Proposal The first mismatch between the service level monitoring and the error budget is the puma apdex component. So in this project we'll introduce a new SLI, defined in the application and consumed in the service monitoring and error budgets. Roughly: 1. Perform the groundwork to define SLIs in the rails project 1. Provide the mechanism to Stage Groups to be able to set their own thresholds per endpoint. Keeping in mind the limitations regarding resource consumption. 1. Consume these in the service catalog as new indicators for the `web`, `api` and `git` fleets. 1. Add documentation on how stage groups can adjust thresholds. 1. Set up a process where Stage Groups can submit changes for review ## Shipping this work We will deliver this feature in two parts. https://gitlab.com/groups/gitlab-com/gl-infra/-/epics/573 : In this epic, we will enable stage groups to set custom SLI's but they will not yet be reflected in the Error Budgets. We expect it will take time for stage groups to set custom SLI's and that there will be some engagement involved. By starting this engagement as soon as possible, we will have SLI's in place by the time we have the ability to incorporate this into the Error Budget. https://gitlab.com/groups/gitlab-com/gl-infra/-/epics/573 : In this epic, we will use the custom SLI's and reflect them in the Error Budgets. |Workstream 1 | Workstream 2| |-------------|-------------| | Deliver https://gitlab.com/groups/gitlab-com/gl-infra/-/epics/573 - Ability to set custom SLIs | | | Work with stage groups to define SLIs | Deliver https://gitlab.com/groups/gitlab-com/gl-infra/-/epics/573 - Incorporate custom SLIs into Error Budget | ### Status 2021-01-21 Everything is in place for stage groups to opt in to the new SLI. For stage groups, the `puma-apdex` component still exists while they are adjusting urgencies. But for service level monitoring, we are fully relying on `rails_requests-apdex`, which is backed by the new metrics. One bug affecting the new SLI (https://gitlab.com/gitlab-com/gl-infra/scalability/-/issues/1500) was discovered thanks to groups opting in and has been fixed. When all groups have opted in at the end of the quarter (hopefully) (https://gitlab.com/gitlab-com/gl-infra/scalability/-/issues/1343), we'll be able to remove the separate recordings for `puma-apdex` used by the error budgets for stage groups (https://gitlab.com/gitlab-com/gl-infra/scalability/-/issues/1481). Next, we're going to improve the observability by building a focused dashboard for this &664