We ran into a number of problems with deploying today: 1. Migrations on blessed worker failed due to a database statement timeout: https://gitlab.com/gitlab-org/gitlab-ce/snippets/1657595 1. The migration, `20170124193205_add_two_factor_columns_to_users.rb`, was left in incomplete state, where the columns existed but the index was invalid. 1. Disabled the statement timeout altogether via `ALTER DATABASE gitlabhq_production SET statement_timeout = 0` 1. We deleted the invalid index, re-created it by hand, and inserted `20170124193205` into `schema_migrations`. 1. Running the deploy again did not re-run the migrations and instead deployed to the whole cluster without checking the migration status. 1. Error 500s abounded since there were missing migrations. 1. Re-ran the migrations by hand, but there were a number of long-running migrations that added default values to both `projects` and `users` table. 1. pgbouncer died in the middle of these migrations (reloaded config?) and caused more Error 500s. 1. There was one post-migration to remove temporary files that took too long (https://gitlab.com/gitlab-org/gitlab-ce/issues/30866). /cc: @felipe_artur, @yorickpeterse, @twk3, @rspeicher, @ayufan, @godfat