Workhorse web canary apdex dips

Summary

We see constant slow burns for the workhorse SLI for canary stage. This ends up paging the on-call with the 6h window on a weekly basis, and we can also see constant dips if we look at the 30 day view:

Screenshot_2022-06-30_at_08.52.32

Source

Looking at the 30 days on this alert they all seem to happen around the same time 14:00 UTC - 18 UTC, which is when we start having peak traffic.

Screenshot_2022-06-30_at_08.57.00

This might indicate that we are under-provisioned in cny and can't handle the load, or potentially have some specific endpoint that is hurting the apdex.

Related Incident(s)

Originating issue(s):

Desired Outcome/Acceptance Criteria

Associated Services