May 10th, 2019 The daily 7:20AM UTC from May 10th, 2019 appears to have broken dev.gitlab.org as the site started returning 502s. Alerts: - https://gitlab.slack.com/archives/C101F3796/p1557473864073300 - https://gitlab.slack.com/archives/C101F3796/p1557474669078500 Following [the runbook](https://gitlab.com/gitlab-com/runbooks/blob/master/howto/dev-environment.md) yielded the following: ``` -rw------- 1 root root 2 May 9 07:27 db-migrate-873248b1f0d3a7a5535771a3a1635803-12e12d9672a -rw------- 1 root root 2 May 10 07:27 db-migrate-873248b1f0d3a7a5535771a3a1635803-84f908fbd69 ``` ``` git log --oneline 12e12d9672a..84f908fbd69 84f908fbd69 Merge branch 'patch-29' into 'master' b3665572d13 Open merge collab link in new tab 8361506afe7 Merge branch 'support-string-piwik-website-ids' into 'master' 6b2f4ea5895 Merge branch '6105-move-ee-differences-group-members' into 'master' 269110b9bbc Merge branch 'acme-account-private-key' into 'master' 444959bfa0b Generate Let's Encrypt private key 81528a3a88f Merge branch 'patch-17' into 'master' ab37a664aed Merge branch 'sh-upgrade-ruby-2.6.3-ce' into 'master' 615e57c7c51 Merge branch 'add-warning-to-backup-rake-task' into 'master' 9ecc701aa5b doc: administration/planutml: update instructions for setting dialogs 16d4eeb217b Merge branch 'antonyliu-i18n-user-profile' into 'master' 8021efb72de i18n: externalize strings from user profile settings a64904e41f3 Merge branch '61596-increase-timeout-for-discussion-notes-spec' into 'master' deb5276f8bd Add warning that gitlab-secrets isn't included 38635c645e5 Adds quotes to Matomo/Piwik website ID 85cf320258a Fix discussion notes spec timeout a50951d73a9 Upgrade Ruby version to 2.6.3 5f54e2e9cb9 CE: Move EE differences for ``` Remy helped find this: https://about.gitlab.com/handbook/engineering/development/enablement/distribution/maintenance/dev-gitlab-org.html#manually-upgradingdowngrading-packages. Hence, creating the issue. ### Proposed Action Downgrade the installed version