Quantifying renovate toil
Problem
We have a large number of renovate issues that (without any research) likely fall into two rough categories:
- Renovate MRs that update dependencies in various projects.
- Renovate MRs that update helm charts in k8s. https://gitlab.com/gitlab-com/gl-infra/k8s-workloads/gitlab-helmfiles/-/merge_requests/?sort=merged_at_desc&state=opened&label_name%5B%5D=dependencies&first_page_size=20
Each of those needs some more data around exactly how much of a problem this is. The reason we need this data is two fold: First, I suspect it is a much larger problem than it seems on the surface. Second, we can use this as a before to compare to for any future improvements.
The data that I think we need is pretty straight forward but not so easy to get.
- How many renovate MRs are there of each type?
- How long do they take to deal with on a regular basis?
- How often do they fail?
As far as the numbers go, simply getting the number of MRs is something that we can handle with some scripts and scraping the GitLab.com API. I'll start working on that soon.
It's 2 and 3 that are a lot harder to get. It would be ideal if we could come up with some estimates on minimum and maximum amount of time that we spend on each MR type, even if they're just looking at a handful of MRs and doing some math based on that.
Failure is much harder, particularly for the helm MRs. @pguinoiseau @mchacon3 @nduff you seem to have done a large number of them, do you have any suggestions on how we can best track failure rates? I'm happy to go dig through whatever data set we have to come up with some answers if we need to, but if you have a better answer, I'm all ears. cc @donnaalexandra as well.
Analysis
Caveat to this analysis: This is only looking at renovate MRs under gl-infra in both location. I believe this should cover most of it, but we don't presently have a good way to target only the projects we own. I'll be addressing that as part of this project.
GitLab.com
Stats
| Month | Number of Renovate MRs | Failure rate | Automerged | Top approver | Top Project |
|---|---|---|---|---|---|
| Oct 24 | 1401 | 4.4% (62) | 42.3% (593) | Pierre (193) | k8s-workloads/gitlab-helmfiles |
| Nov 24 | 1125 | 4.4% (49) | 37.5% (422) | Pierre (274) | k8s-workloads/gitlab-helmfiles |
| Dec 24 | 868 | 2.6% (23) | 42% (325) | Pierre (213) | k8s-workloads/gitlab-helmfiles |
| Jan 25 | 1052 | 2.4% (25) | 34.5% (363) | Pierre (216) | k8s-workloads/gitlab-helmfiles |
| Feb 25 | 1377 | 3.5% (48) | 33.2% (457) | Pierre (377) | k8s-workloads/gitlab-helmfiles |
| Mar 25 | 1592 | 4.8% (77) | 32.2% (513) | Andrew (303) | k8s-workloads/gitlab-helmfiles |
Top projects for March 2025 (more than 30 MRs):
| Project | Count | Percentage |
|---|---|---|
| k8s-workloads/gitlab-helmfiles | 212 | 13.3% |
| sandbox/switchboard_uat | 166 | 10.4% |
| gitlab-dedicated/instrumentor | 76 | 4.8% |
| runway/runwayctl | 40 | 2.5% |
| us-public-sector/switchboard_pubsec | 36 | 2.3% |
| gl-infra/infra-roadmap | 36 | 2.3% |
| sandbox/switchboard_runners | 35 | 2.2% |
| gl-infra/common-ci-tasks-images | 35 | 2.2% |
| gl-infra/ci-images | 33 | 2.1% |
| gl-infra/common-ci-task-tests | 32 | 2.0% |
Ops
Stats
| Month | Number of Renovate MRs | Failure rate | Automerged | Top approver | Top Project |
|---|---|---|---|---|---|
| Oct 24 | 78 | 2.6% (2) | 10.2% (8) | Pierre (66) | gl-infra/config-mgmt |
| Nov 24 | 68 | 2.9% (2) | 7.3% (5) | Pierre (63) | gl-infra/config-mgmt |
| Dec 24 | 60 | 6.7% (4) | 11.7% (7) | Pierre (48) | gl-infra/config-mgmt |
| Jan 25 | 99 | 5.1% (5) | 16.2% (16) | Pierre (77) | gl-infra/config-mgmt |
| Feb 25 | 138 | 10.1% (14) | 23% (32) | Pierre (88) | gl-infra/config-mgmt |
| Mar 25 | 169 | 6.5% (11) | 41.1% (70) | Pierre (80) | cells/tissue |
Top projects for March 2025 (more than 5 MRs):
| Project | Count | Percentage |
|---|---|---|
| cells/tissue | 73 | 43.2% |
| gl-infra/config-mgmt | 69 | 40.8% |
| k8s-mgmt/fleet | 5 | 3.0% |
Overall totals
- Total number of MRs handled in both environments over the last six months: 8027
- Total number of MRs handled by a human (mostly named Pierre) in both environments over the last six months: 5216 (65%)
- Literally just Pierre's approved MRs: 1961 (24.4% of total, 37.5% of human managed MRs)
Totals broken up roughly by team for March
- Runway: 97 (projects called runway)
- Dedicated: 538 (projects called dedicated, switchboard or sandbox)
- Observability: 112 (projects called observability or gitlab-com)
All the rest roughly falls between teams and has no real owner, so mostly falls to Foundations to approve (as demonstrated by Pierre's count above)