Intermittent apdex dips across multiple services, primarily web.
In gitlab-com/runbooks!3214 (merged), the apdex SLO target for the webservice was unfortunately dropped from 99.9% to 99.8%. This was done because the previous threshold is no longer being met and was leading to unhelpful alerting messages.
Since the beginning of January 2021, the apdex score for the web has trended down, which is why we've had to adjust the SLO:
# Possible Causes
Looking at our rails services, the apdex degradation appears to come in waves.
For example, from the web Puma SLI Apdex:
However, these drops are not only limited to web. We see them in git and api too.
API Puma SLI Apdex
Git Puma SLI Apdex
