Loading
Add SLI for Rails to OpenBao calls
Why this MR ?
- Every Secrets Manager secret read, write and permission check crosses one HTTP hop from Rails to OpenBao
- We already have three Prometheus metrics on that hop, but they are plain counters and a histogram, so nothing can alert on them and they never reach our stage group error budget
- The documented way to get both is an application SLI - https://docs.gitlab.com/development/application_slis/
- Covers part of #611251
What does this MR do ?
- Adds the
openbao_client_callsapplication SLI and records it from the same place the existing metrics are recorded - Error rate only, no apdex - this hop's latency is dominated by OpenBao's four synchronous audit POSTs back into Rails, so any target worth setting would be red from day one
- The existing duration histogram still carries the percentiles
- Documents the two new counters
Notes for review
- The SLI module sits in
lib/rather thanee/lib/becauseGitlab::Instrumentation::Openbaois already a FOSS file and is the call site, and a FOSS file cannot reference an EE constant
References
- Issue - #611251
- MR that added the raw metrics - !251497 (merged)
- MR that fixed their outcomes - !252512 (merged)
- Runbooks MR that consumes this SLI - gitlab-com/runbooks!11333 (merged)
- Application SLI docs - https://docs.gitlab.com/development/application_slis/
Edited by Jayakrishnan Mallissery