Checklist for running incident scripts in production
Context
Oftentimes when we have an incident that results in the need to do some data cleanup via a Rails console, a developer with subject matter expertise will write a script that can be used to perform this and then it gets handed over to the SREs to be ran.
It would be helpful to have a checklist of things to check for when running these scripts.
Example
An example from a recent incident:
.
for chained Ruby statements
Appending When ruby statements are being chained, they are oftentimes broken onto separate lines for readability. It might look something like this:
user = User
.joins(:something)
.where('some_statement')
.where('some_other_statement')
Unfortunately this can cause problems when running a rails console unless we change the .
to be appended like so:
user = User.
joins(:something).
where('some_statement').
where('some_other_statement')
-
Make sure any chained Ruby statements have the .
appended instead of prepended
A few questions
- Is there enough content for something like this?
- Where should this live?
- Perhaps in the incidents directory?
- As part of an existing file or elsewhere?
- Should we add it to the checklists in the README?
- What can we do to make sure people who need this can find it?