2020 in numbers: GitLab.com deployment and Delivery team overview
At the end of the year 2020, it is a good time to create an overview of the year behind us. With the main Delivery team epic called Release velocity, lets start an overview with the deployment overviews.
GitLab.com deployments in 2020
In 2020, we deployed to GitLab.com approximately 433 times
This is the distribution per month:
| 2020 | Number of deploys |
|---|---|
| January | 21 |
| February | 19 |
| March | 15 |
| April | 28 |
| May | 28 |
| June | 32 |
| July | 37 |
| August | 39 |
| September | 57 |
| October | 57 |
| November | 45 |
| December | 55 |
It is remarkable (albeit logical) to see how the increase in number of deployments had a significant positive impact on Mean time to Production
Gsheet, Grafana annotations source . Note The source is collected from Grafana dashboard annotations which is the only dataset that has not changed significantly through the year.
Collecting from another source, gitlab-org/gitlab deployment data (feature expanded by the Delivery team with external environments feature) shows a similar trend even if not exactly the same numbers. This is expected as we have had issues periodically with reliably sending the data between ops.gitlab.net and GitLab.com.
In the table above you can also see the number of deployed MR's from gitlab-org/gitlab repository within the deployments we shipped, remarkable numbers!
One of the pivotal moments for the above dashboards was the change in the deploy cadence to daily. While this might seem like a simple thing from this perspective, the team had to invest significant time into automating manual steps in order to be able to keep with the volume of work. One of the items that significantly decreased the pressure on the team was the changes in security releases, covered more in the next section.
Releases for self-managed
Another responsibility that not any less important are releases for self-managed users. In addition to the monthly release published on the 22nd, we release patch (bugfix) releases as well as security releases.
In 2020, we completed approximately an average of 10 releases per month
One remark here is that month of December is missing 2 releases since the data warehouse has not been updated in over 2 weeks.
One of the most impactful changes done in the releases for self-managed was the Security release process scaling. This was not only impactful for the Delivery team, but it had vast positive consequences for Development, and Security departments. From moving the security development process to GitLab.com at the very beginning of the year, other impactful improvements, to (almost) immeasurable impact that removing the blocking nature of security releases had on GitLab.com deployment velocity. The last one gave us an opportunity to reduce the MTTP target to 12 hours in September, which was previously at 24 hours.
GitLab.com Kubernetes migration
In 2020 we ramped up with the work on our migration of GitLab.com to Kubernetes. I'll start this overview with something significant that is easily overlooked. We have not slowed down any of our day to day actions with deployments while migrating our first service to K8s that needed to be in sync with our VM deployments!
We also moved away from a single cluster, to a total of 4 clusters in production giving us more flexibility and significant cost saving of $20k USD per month we would otherwise have to spend. This is not to mention the fact that we could remove a fleet of NFS servers, isolating remaining NFS nodes to only GitLab Pages and a small set of sidekiq nodes that need to serve it.
Sidekiq migration which was significantly boosted by the work that ~"team::Scalability" did on the application side, was a large undertaking that had a large impact on the deployment speed of that service and the deployment as a whole. I am unfortunately unable get the real exciting numbers such as, the number of nodes we currently have running in production to compare with the fleet size at the beginning of the year due to not having access to the required environments (such as production gcp account). It would be interesting to see the number of requests we served at the start of the year, compare it with the end of the year and make some conclusions on how much more scaleable we are as a platform. In any case, I would be surprised and honoured if you actually read everything up until this line, and if you did you have my thanks.
What is in line for 2021?
This issue came as a prep for writing out a strategy for the team in 2021, so the first thing in line is setting that direction and placing it in the handbook, all aligned with the department strategy.
Currently top of mind are:
- Get better at collecting metrics such as the ones I created above. We have a lot of data across a number of platforms, and we have to find a way to make it simpler to collect. These metrics will inform a lot of our future decisions.
- Migrate all remaining stateless services to Kubernetes. Start planing stateful service migrations.
- Change the team direction towards shipping GitLab features and replacing custom tooling given that the 12 hour MTTP target is a good turnaround time for majority of stakeholders.
- Plan for an impact on release policies due to additional company ambitions (vague on purpose)
Happy 2021