Find the root cause of the Configure high apdex api error spend
Goal
Fix our error budget
Proposal
- Find the root cause
✅ - Fix it or increase the limit
The main culprit is the /api/:version/projects/:id/terraform/state/:name endpoint. As decryption happens in the Rails app and Terraform state file sizes grow, the 1s limit is unreasonable. We should increase the limit to 3s.
The 2nd culprit is /api/:version/projects/:id/terraform/state/:name/lock where optimisation is needed.
Root cause preliminary insights
As shown on this chart, most of our spend is related to apdex and api type errors in puma
apdex: This means an operation that succeeded, but did not perform within the set threshold.
Puma Apdex: slow requests
This shows the number of requests exceeding 1s request duration per endpoint over the past 7 days.
This is the only threshold at the moment. We're discussing making it configurable in this issue. Let us know if you know of an endpoint that would need this kind of customization.
A quick search on https://log.gprd.gitlab.net/app/discover#/ using the query json.duration_s > 1 and (json.meta.feature_category : kubernetes_management OR json.meta.feature_category : secret* OR json.meta.feature_category : infrastructure_as_code)