Prevent checking in allow_stopping_for_update=true in terraform
Summary
By default allow_stopping_for_update
is false in terraform to prevent rebooting nodes by accident without warning when executing seemingly safe terraform plans. We usually manually set allow_stopping_for_update = true
when tf apply
refuses a change because of a needed reboot. But often we then also permanently check in allow_stopping_for_update = true
, which removes this extra safety. This caused production#3578 (closed).
We should have some CI checks to prevent this or at least warn when checking that in. Or maybe use env VARs to be able to override this in CI pipelines.
Related Incident(s)
Originating issue(s): production#3578 (closed)
Desired Outcome/Acceptance criteria
- Fail the Terraform CI process when
allow_stopping_for_update = true
is detected in the changes. - (Optionally) allow a CI pipeline variable to be set to allow overriding the default
allow_stopping_for_update = false
to true for a single pipeline.
Associated Services
Corrective Action Issue Checklist
-
link the incident(s) this corrective action arose out of -
give context for what problem this corrective action is trying to prevent from re-occurring -
assign a severity label (this is the highest sev of related incidents, defaults to 'severity::4') -
assign a priority (this will default to 'priority::4')
Edited by Cameron McFarland