Tool for executing and reverting Rails migrations on staging
We need a tool that can execute a Rails migration (or multiple) from a merge request using the staging database, gather some statistics (timings, etc), then revert the changes. This would allow endbosses to easily assess the impact of database migrations. This tool would take an MR URL as input, SSH into a staging worker, download the migrations (and only the migrations), and run them using a local Rails process. To revert the changes we'd wrap the migration in a transaction that rolls back automatically (this can be done by raising
To prevent migrations from getting in each other's way we'd have to ensure that only one such process can run at a given time, regardless of the worker that it's running on. We can just use
Gitlab::ExclusiveLease for this (make sure to release it once the process is done).
The data we'd gather should be (if possible):
- The time it took to run the migration
- The output of the migration
If possible we should also gather the number of affected rows and what not, but I don't think there's an easy way of getting this.
Once this script is in place we should hook this up to Marvin. The idea is that an endboss can run something like:
@marvin: check migrations for https://gitlab.com/gitlab-org/gitlab-ee/merge_requests/915
Marvin would then call the script and simply dump the output in the channel. This removes the need for giving all endbosses SSH access to our staging workers. It also ensures that if the results are fishy there are multiple people noticing it (= everybody in the channel).
We can not hook this up in CI. Doing so would allow naughty "contributors" to run arbitrary Ruby code using our staging database. This has a 100% guarantee of being exploited in the worst ways imaginable.