Determine cause of 12 min interval spike in HTTP Queue Timing - and solve it.
Update
Moved the original intent of the issue (alert on high HTTP queue timing) to https://gitlab.com/gitlab-com/infrastructure/issues/2667. Now using this issue to track progress on identifying why there is a spike in HTTP queue timings every 12 minutes.
Original title: Alert when web/api/git fleets are consistently over 60% CPU usage
This comes from a previous experiment (https://gitlab.com/gitlab-com/infrastructure/issues/2058) where we determined that whenever one of these fleets were over 60% CPU usage we started seeing an impact in the form of HTTP queueing timings increasing due to not having enough front end capacity.
To prevent this from happening we should alert ourselves in the form of a warning to consider upscaling the fleet.
This, of course, can be completely ditched the moment when we can make this happen with auto-scaling, as this will be an automatic behavior.