DRI @alejandro ## Background The Gitaly fleet requires regular upgrades. The process is not well defined and there is not much automation available. In order to not make this epic confidential, I cannot relay the entire history of this work here. Please see https://gitlab.com/gitlab-com/gl-infra/infrastructure/-/issues/10039 for more details. ## Work Required The goal of this epic is to prepare the process and automation to simplify the work required to upgrade a gitaly server, followed by performing the upgrade itself. Broadly, it involves the following steps: - Create a process to build packer images baked with a given omnibus-gitlab version installation https://gitlab.com/gitlab-com/gl-infra/scalability/-/issues/1498 - Develop automation to speed up the process and reduce the possibility of human error https://gitlab.com/groups/gitlab-com/gl-infra/-/epics/602 - Develop a Change Request plan to identify all the necessary steps for each upgrade batch https://gitlab.com/gitlab-com/gl-infra/scalability/-/issues/1382 - Perform the Change Requet process on gstg to validate it https://gitlab.com/gitlab-com/gl-infra/scalability/-/issues/1421 - Coordinate a maintenance window to perform the upgrade on gprd https://gitlab.com/gitlab-com/gl-infra/scalability/-/issues/1423 - Perform the change on gprd https://gitlab.com/gitlab-com/gl-infra/scalability/-/issues/1425 ## Upgrades Completed | Batch | Name | No. servers | Completed date | |-------|------------------------|-------------|-| | 0 | canary | 1 | `2022-03-02` | | 1 | part single zone | 4 | `2022-03-03` | | 2 | hdd | 4 | `2022-03-03` | | 3 | marquee | 4 | `2022-03-03` | | 4 | pre 2019 | 4 | `2022-03-03` | | 5 | post 2019 | 4 | `2022-03-03` | | 6 | HDD (4), Multizone (20) | 24 | `2022-03-04` | | 7 | HDD (2), Multizone (17) | 19 | `2022-03-04` | | 8 | Rest of single-zone (18) | 18 | `2022-03-04` | | 9 | Praefect (1) | 1 | `2022-03-04` | | 10 | Praefect (1) | 1 | `2022-03-04` | | 11 | Praefect (1) | 1 | `2022-03-04` | ### Notes from performing the upgrade - The original maintenance period of 1 hour was too small. Even though we gained confidence enough to run larger batches, the small window increased the stress of the process. For future maintenance, a 2-hour window is preferred. - During the upgrade call, we had 1 SRE running the scripts, 1 SRE on hand to look into specific failures, and 1 additional engineer who was interpreting graphs and describing the situation as the servers went offline and came back online. This should be the minimum required for future upgrades. - During the call we also had at least one EM (IM) available to monitor progress, coordinate with the CMOC, and keep people informed as to what was happening. - Keeping Praefect separate meant that we could concentrate on which problems were specific to the OS upgrade, and which problems were specific to Praefect. ## Status 2022-04-20 The OS upgrade is complete on the production servers. 85 servers were upgraded over 3 days. Cleanup tasks are done. The process has been documented for future upgrades, so that it can be executed by the relevant teams when needed.