Tool for executing and reverting Rails migrations on staging

We need a tool that can execute a Rails migration (or multiple) from a merge request using the staging database, gather some statistics (timings, etc), then revert the changes. This would allow endbosses to easily assess the impact of database migrations. This tool would take an MR URL as input, SSH into a staging worker, download the migrations (and only the migrations), and run them using a local Rails process. To revert the changes we'd wrap the migration in a transaction that rolls back automatically (this can be done by raising ActiveRecord::Rollback).

To prevent migrations from getting in each other's way we'd have to ensure that only one such process can run at a given time, regardless of the worker that it's running on. We can just use Gitlab::ExclusiveLease for this (make sure to release it once the process is done).

The data we'd gather should be (if possible):

• The time it took to run the migration
• The output of the migration

If possible we should also gather the number of affected rows and what not, but I don't think there's an easy way of getting this.

Once this script is in place we should hook this up to Marvin. The idea is that an endboss can run something like:

@marvin: check migrations for https://gitlab.com/gitlab-org/gitlab-ee/merge_requests/915

Marvin would then call the script and simply dump the output in the channel. This removes the need for giving all endbosses SSH access to our staging workers. It also ensures that if the results are fishy there are multiple people noticing it (= everybody in the channel).

We can not hook this up in CI. Doing so would allow naughty "contributors" to run arbitrary Ruby code using our staging database. This has a 100% guarantee of being exploited in the worst ways imaginable.

cc @pcarranza