## Background [GitLab Workhorse](https://gitlab.com/gitlab-org/gitlab-workhorse/) is a GitLab component that fills a niche that is not well served by the main Rails web application: slow requests. Its best known features are to enable `git clone` via HTTP and handling large file uploads and downloads. Although Workhorse is tightly coupled to Rails it lives in its own project [gitlab-org/gitlab-workhorse](https://gitlab.com/gitlab-org/gitlab-workhorse/). This was a natural choice when Workhorse was created but it is now a source of friction. 1. It is bad for contributors: feature development across two repositories is harder than doing it in one repository, and almost all development that involves workhorse also involves the main Rails repository 1. It is bad for the Workhorse maintainers: having a separate project can sometimes be a good trade-off for a team, where they pay the price of maintenance overhead in return for velocity. But Workhorse has no team, just a small group of maintainers and reviewers from different teams and departments (Development and Infrastructure). With such limited resources the overhead becomes more onerous and it is unclear who benefits from the velocity. ## Proposal We want to transition Workhorse development from its current isolated project to the main repository [gitlab-org/gitlab](https://gitlab.com/gitlab-org/gitlab/). At the end of the transition the Workhorse code is developed and released as a sub-directory `/workhorse` of the main repository, and the main issue tracker is used for Workhorse issues. The old repository gets switched to read-only. ## Benefits 1. It will be easier to develop and test Workhorse features 1. It will no longer be necessary to tag Workhorse releases, which makes shipping Workhorse changes faster 1. In particular, security releases will be much less complicated then they are now: no more non-public workhorse security tags etc. 1. Workhorse documentation will live in the main documentation tree and can benefit more easily from review by our technical writers 1. It will be easier for newcomers to notice that Workhorse exists, and for everybody, to grasp what it does ## Plan ### :white_check_mark: Phase 1: dual releases The current release process involves updating the version number in the `GITLAB_WORKHORSE_VERSION` file in the main repository. The outcome of phase 1 is that every time we update `GITLAB_WORKHORSE_VERSION`, we also update a copy of the Workhorse source code that lives in the main repo. We can have this copy be a "Git subtree" meaning we retain all Workhorse commit history. 1. Decide on what subdirectory of the main repo workhorse will live in 1. Create a script in the main repo that reads `GITLAB_WORKHORSE_VERSION` and updates the embedded copy of gitlab-workhorse to that version 1. Add a CI check in the main repo that makes sure the embedded copy of gitlab-workhorse matches `GITLAB_WORKHORSE_VERSION` ### :white_check_mark: Phase 2: update consumers to fetch Workhorse from the main repo After phase 1 we know that fetching Workhorse by tag based on `GITLAB_WORKHORSE_VERSION` or compiling the embedded copy gives the same result. We can then gradually update all consumers of the Workhorse source code (source installations, Omnibus, CNG, the GitLab test suite) to build Workhorse from the Rails-embedded copy instead of the stand-alone repository. We continue to have dual releases throughout phase 2. GDK keeps using the standalone repo because the embedded copy of workhorse is still read-only. Note that we do not have to change any process supervision code (Runit, init script etc.) because we can keep installing the Workhorse binaries in the same place they were in before. All we need to change is the code that produces a Workhorse binary. 1. :white_check_mark: Make sure that the embedded copy of gitlab-workhorse compiles. (This may need creative solutions with `go.mod`?) https://gitlab.com/gitlab-org/gitlab/-/merge_requests/48690 1. :white_check_mark: In the main repo, convert `rake gitlab:workhorse:install` to use the embedded copy https://gitlab.com/gitlab-org/gitlab-workhorse/-/issues/334 1. :white_check_mark: Convert Omnibus so that it builds Workhorse from within `config/software/gitlab-rails.rb` and remove `config/software/gitlab-workhorse.rb` https://gitlab.com/gitlab-org/gitlab-workhorse/-/issues/332 1. :white_check_mark: Update CNG so that it compiles workhorse from the main repo https://gitlab.com/gitlab-org/gitlab-workhorse/-/issues/333 1. :white_check_mark: Update source installation and upgrade documentation to compile from the embedded copy (target directory should remain `/home/git/gitlab-workhorse` to avoid init script changes) https://gitlab.com/gitlab-org/gitlab-workhorse/-/issues/334 1. :white_check_mark: Have CI in the main repo run `make test` on the embedded copy of workhorse https://gitlab.com/gitlab-org/gitlab/-/merge_requests/47061 https://gitlab.com/gitlab-org/gitlab-workhorse/-/issues/335 ### Phase 3: stop using the old repository 1. :white_check_mark: Migrate all open Workhorse issues to the main issue tracker -> too many for manual work, must write API script https://gitlab.com/gitlab-org/gitlab-workhorse/-/issues/355 1. :white_check_mark: In the README, make clear MR's and issues should go to gitlab-org/gitlab. https://gitlab.com/gitlab-org/gitlab-workhorse/-/merge_requests/683 1. :white_check_mark: Close all open MR's on the old repo and ask for them to be resubmitted on gitlab-org/gitlab -> can be done manually, not many MR's 1. :white_check_mark: Update GDK to use gitlab/workhorse https://gitlab.com/gitlab-org/gitlab-development-kit/-/merge_requests/1808 1. :white_check_mark: Update README in new location with canonical source https://gitlab.com/gitlab-org/gitlab/-/merge_requests/54941 Stop making dual releases. Switch GDK to using the embedded copy of workhorse. ## Open questions 1. Q: What subdirectory should Workhorse be in? `/go/gitlab-workhorse` is maybe too much about Go. Other suggestion is `/components/gitlab-workhorse`. https://gitlab.com/groups/gitlab-org/-/epics/4826#note_442471817. - A: Let's use `/workhorse`. 1. Q: Do we need to worry about `go.mod` [major version gotchas](https://blog.golang.org/v2-go-modules#TOC_3.) once we start having Workhorse 13.x as opposed to 1.x? https://gitlab.com/groups/gitlab-org/-/epics/4826#note_443019400. - A: JV: I don't think this will be a problem because we never install workhorse with `go get`. 1. Q: how would this impact licensing? Workhorse is now in a stand-alone repo with an MIT license. After we move it it would fall under the license of the gitlab-org/gitlab repo. - A: It looks like as long as we don't put workhorse in an `ee/` subdirectory, it will stay MIT.