Cloudwatch CPU alarms firing for CI runner - PagerDuty

When we run Gitlab CI jobs on the ci-runner-01 box, it is firing cloudwatch CPU alarms which triggers PagerDuty

Examples

• https://gitter.pagerduty.com/incidents/P3VWDE5
• https://gitter.pagerduty.com/incidents/PRQTFI8

Relevant code

• https://gitlab.com/gitlab-com/gl-infra/gitter-infrastructure/blob/fe8a88ace05365399d8db27c0d0c9f00334b4bab/ansible/roles/cloudwatch-alarms/tasks/main.yml

Remediation

• Tone down the alarms to be 95% CPU for an hour

cc @viktomas