Alert on high 503 error rate
For production#5521 (closed), the degradation started at 10:00 UTC but we were not aware of the problem until customers started reporting issues 3 hours later.
Since requests were not making their way to the api fleet, our SLO metrics didn't complain. Hence, we need to alert on high 503 error rate, which could indicate a problem with the intermediate components between HAProxy and Rails.