Workhorse Ownership
Background
To over-simplify the history, Workhorse was created ~5 years ago primarily to handle git
operations via HTTP(S) since these operations can sometimes take an extraordinarily long time. For this reason, it made sense that Workhorse would be owned by devopscreate. Later, devopscreate split into new groups, and it went with groupsource code.
Workhorse ownership has been discussed many times since it's creation[1][2][3], since it quickly became responsible for many other things - like object storage, images, CI long polling, error tracking, and so on.
Problem
The problem to solve is once again ownership. ~"group::memory" has been working on a large effort towards dynamic image resizing for avatars and content images which would have benefited from having a dedicated team. The maintenance this work will require in the future would also benefit from a dedicated team. This is just one example, but groupsource code is not able to give Workhorse the eyes and routine work that it needs.
Data
MRs created over time | Issues created over time |
---|---|
The number of merge requests created in the project has steadily increased over time, while the number of those being from groupsource code have become inconsistent or declined. This signifies a branching of responsibility. | Issues created over time VS being closed have enough of a gap for groupsource code to be required to work on this project each release, while there is no capacity from the team to do so. |
Proposal
Originally, my proposal would have been to find a new team to take ownership, or preferably create a dedicated team to maintain Workhorse. However, two things always rang true:
- It doesn't quite fit with any existing teams responsibilities
- The majority of Workhorse work is coming from particular teams who need it
Assuming we cannot find a solution to 1. (finding a new team that it mostly fits with), I am suggesting that we treat Workhorse the same as we do with GitLab-rails - we have a diverse set of maintainers on the project, but different teams have ownership in the project.
One consideration with that approach is what do we do with more generic, or global, changes that need to happen in Workhorse, like redesigning content headers? This is a problem in GitLab-rails today as well. For this I can offer 3 suggestions:
- The team who needs the work done first, or is affected the most, should schedule the work (happens in GitLab today - suggest going with this approach to see how often this occurs)
- The team who is most experienced in the particular area schedules the work (probably hard to determine, since ownership in general has been hard to determine)
- The individuals who are most experienced with Workhorse in general or globally will schedule the work (new idea, hear me out, more data below)
More data
Highest number of contributions by author since 2020-01-01 |
---|
I have only included authors who submitted > 5 MRs this year. I expect to see 2 names in this list because they are part of groupsource code and are responsible for Workhorse today. @mkaeppler is currently working on a large feature in the project, so he is expected as well. The other names in this chart are Staff+ engineers who have broader, cross-team experience than a Senior or below in any team would have. |
To be clear, I am not suggesting that one of these people should be responsible for generic Workhorse issues. Instead, I am shining a light on the coincidence that Staff+ roles seem to have made a place in Workhorse and draw the correlation back to Staff responsibilities - can we consider the Workhorse issues that are outside of an individual team to be of a broader impact to GitLab as a whole that a Staff+ role would take the initiatives on?
Pros and Cons for Team Ownership
Proposed Team Ownership | Pros | Cons |
---|---|---|
Source Code (@nick.thomas to check for accuracy) | * Technically the team with the most knowledge about Workhorse in general | * Team itself has less "stake" in Workhorse functionality / Source Code features do not align with Workhorse |
Enablement (@craig-gomes to check for accuracy) | * Already has teams dedicated to "concepts" that exist across the devops stages | * Not as clear of a relationship to user impact and revenue. With Enablement taking 14% of our R&D spend already, the extra investment will require a business case. |
Scalability (@rnienaber to check for accuracy - done) |
* We already have a Workhorse maintainer in the team * Workhorse is in essence a component that enables us to take slower operations away from gitlab-rails. |
* Workhorse scales up gitlab-rails so that it can handle slow things. This is orthogonal to gitlab.com's scaling challenges, which are about doing many things. * If a new feature is proposed, it will be prioritised in accordance with how it will impact GitLab.com (self managed customers are not Infrastructure's primary concern). * The other two backend engineers in the team already have non-team duties in the form of being GitLab BE Maintainers. |