2021 in numbers: GitLab.com deployment and Delivery team overview
As 2021 comes to a close it's a good time to stop and reflect on the year. Compared to 2020 Delivery have some new metrics as part of our work to increase visibility into MTTP. Let's take a look.
GitLab.com deployments in 2021
In 2021 we deployed approximately 686 times, an average of 57 times per month. This is an impressive increase of 58% compared to our monthly average of 36 in 2020.
| 2021 | Number of Deploys |
|---|---|
| January | 46 |
| February | 46 |
| March | 53 |
| April | 58 |
| May | 42 |
| June | 48 |
| July | 58 |
| August | 80 |
| September | 62 |
| October | 79 |
| November | 66 |
| December | 48 |
The increase in deployment numbers is a result of improved deployment tooling to reduce waiting, changes to the team to begin deploying additionally in APAC hours, and progress on the GitLab.com migration to Kubernetes reducing deployment times (more details below).
It's great to see that the work to deploy more frequency again had a positive impact on Mean Time To Production
The number of deployed MRs is difficult to accurately count because of the number of repos involved in deployments and some inconsistency in labeling but even as an indication we can see we're deploying an impressive number of changes
Sisense
Similar numbers are also found using the MR search, extended by Delivery to support search for deployed changes by environment - gsheet
| Rails | Gitaly | Total MRs | |
|---|---|---|---|
| Jan-2021 | 1603 | 98 | 1701 |
| Feb-2021 | 2206 | 116 | 2322 |
| Mar-2021 | 2303 | 98 | 2401 |
| Apr-2021 | 2489 | 119 | 2608 |
| May-2021 | 1921 | 108 | 2029 |
| Jun-2021 | 2136 | 108 | 2244 |
| Jul-2021 | 1797 | 95 | 1892 |
| Aug-2021 | 1790 | 74 | 1864 |
| Sep-2021 | 1895 | 71 | 1966 |
| Oct-2021 | 1703 | 103 | 1806 |
| Nov-2021 | 1736 | 65 | 1801 |
| Dec-2021 | 1668 | 89 | 1757 |
Going deeper in MTTP
In 2021 we started to look deeper into the factors that affect MTTP. The frequency of deployment is a big factor, but deployment duration and delays from events like incidents and broken Master also impact. We're now deploying frequently enough across all timezones that we need to remove some of the other delays in order to reduce MTTP.
A newly introduced Deployment SLO metric tracks deployment pipeline duration variability to help identify events that lead to longer deployments. With a current target of 8hrs for deployment to all environments, we're seeing around 87% of pipelines meet the target.
To better understand and remove delays to deployment pipelines we started tracking Deployment Blockers in March 2021. From the numbers we have, we see deployments being blocked for anywhere from 8hrs a week right through to 96hrs depending on incidents (including deployment-related failures) and Production Change Locks.
Most interestingly when switched to a monthly view we see the spikes and dips on the deployment blockers chart from weeks 30 through to 45 match up with data for August through to December on the MTTP chart. This indicates that continuing to focus on reducing blockers will give improvements to MTTP.
Releases for self-managed users
2021 has also been a busy year for releases to self-managed users. A total of 97 bug fixes (patch) and security releases have been completed. September was our busiest month with releases being made on 8 separate dates.
| 2021 | Number of Releases |
|---|---|
| January | 9 |
| February | 9 |
| March | 11 |
| April | 7 |
| May | 3 |
| June | 8 |
| July | 11 |
| August | 11 |
| September | 9 |
| October | 6 |
| November | 7 |
| December | 6 |
GitLab.com Kubernetes migration
2021 was another year of great progress on the GitLab.com migration to Kubernetes. The API, Web-fleet, WebSockets, and Pages were all successfully migrated allowing more Virtual Machines to be removed - giving us a maintenance saving as well as a big speed increase on deployments. Many of the increased deployments from July onwards as a result this shorter deploy time.
In December 2021 we migrated the Pages service, and with that also our final stateless service, a huge milestone!
Rollbacks for GitLab.com
One significant achievement that doesn't show up in our metrics is rollbacks. In 2021 we built and tested tools to allow gitLab.com to be rolled back to a previous package. A Rollbacks Check assesses the installed package for suitability and if safe guides the release manager through the process. Several production incidents have already been mitigated in this way and with current rollbacks taking around ~1hr it's a massive time saving compared to the ~6hours needed to deploy a revert MR.
What's coming up in 2022?
In 2022 we'll be continuing to reduce MTTP and improve tooling to automate more of our deployment and release processes. At a high-level we'll be focusing on:
- Reducing the time and involvement needed for patch and security releases. Both are intensive tasks but both are critically important for our users. We plan to increase automation to improve things for both release managers and developers.
- Continuing with the migration to Kubernetes. With the stateless services complete we'll be planning and executing the stateful service migrations. Our first stateful service, Redis is already in progress.
- Focus on adopting and contributing GitLab features to replace our custom tooling. We had some success in 2021 but there is still a lot to do. We'll be switching to focus on this as a major goal for 2022.
- We've made a lot of progress with MTTP over the past few years but we're not done yet. With rollbacks in place, we can take the steps to enable fully automated deployments.
More details on these items, and on how Delivery will develop within Platform will be shared separately.