# Summary Gitaly is the single source of truth for which Git version we are using at GitLab on the server side. Following our upstream-first approach, we typically deploy tagged Git releases to our production systems. Potentially, these tagged Git releases will contain backported patches in case those patches have already been applied upstream. This process has a couple of shortcomings: - The time-to-production between an upstream change in Git and it being usable in Gitaly is up to 4-5 months when accounting for a full Git release cycle. This impacts our ability to iterate fast and close in on developed topics. In case a rollout of a newly developed feature surfaces issue we will be blocked for another full release cycle until we can roll out such a feature and make use of it. - The backporting process is manual and only scales to at most a couple handful of topics that we're upstreaming. As such, we tend to be very conservative when it comes to backporting changes. - The upgrade process is manual, quite heavy weight and takes multiple release cycles due to the requirement to use feature flags for rollback. This wastes engineering time that could be used better for other tasks. - As the upgrade process tends to roll out accumulated changes from 3 months of development it is hard to figure out where specific issues come from in case the rollout causes regressions. This makes it especially hard to figure out performance regressions. As such, this epic proposes to start deploying Git from the `master` branch. This helps to iterate faster, connect issues with their root causes more directly and removes the manual upgrade process. ## Rationale The Git project uses three main branches during development: - `seen` is a collection of in-flight topics that have been sent to the Git mailing list. These series may be in various different states of quality and are still in review. - `next` contains topics that have been cooking for a while and that have been reviewed by multiple Git contributors. These topics have undergone scrutiny of the Git community and are deemed ready for the next release. - `master` contains topics that have been cooking for at least a week in `next` without surfacing any additional testing. The `next` branch undergoes extended scrutiny as many developers of the Git community use this branch. As such, most regressions are catched before they hit the `master` branch. Furthermore, all branches undergo testing via CI on Linux, macOS as well as Windows, which should provide significant test coverage and reassurance that topics are sound once they hit `next` and `master`. As such, the Git project considers the `master` branch to always be ready for release. In addition, the Git project tags releases directly on the `master` branch. There is no specific process in place that would label the `master` branch as release ready. While the Git project doesn't accept new topics in the two weeks before the release, the tagged commit is almost as good as any other typical commit on the `master` branch. As such, the risk to deploy from `master` directly shouldn't be significantly higher than the risk to deploy from a tagged release. Last but not least, the Git project is very strict about backwards-compatibility. Major releases with backwards-incompatible changes have historically been rolled out once per decade. We thus shouldn't have to worry about a deployment causing (intentional) backwards incompatibility with clients and our current usage of Git. ## Safety guards Even so, deploying from `master` requires a couple of additional safeguards. These safeguards include: - Improved dogfooding processes at GitLab to have developers use the `master` branch of Git during day-to-day work. This could for example be realized via the GDK and would ensure that we have increased test coverage of Git. Other companies like Microsoft and Google already do this -- some companies even deploy from `next` directly. - Improved QA processes so that QA knows to always test against the `master` branch of Git ahead of time. - Improved benchmarking for Gitaly so that we can detect performance regressions ahead of time. The Git team has already invested resources to have a [continuous benchmarking process](https://bencher.dev/console/projects/git/plots). - A built-in lag of one or two weeks. We wouldn't deploy the current commit from the `master` branch, but instead the commit from one or two weeks ago. This would ensure that we have enough time to detect and address breakage via our dogfooding process as well as QA and benchmarking. - Gitaly would always deploy two versions of Git: the current version from `master` (with the described lag) as well as the preceding version. These versions can be toggled via a persistent feature flag so that a rollback to a previous version can happen instantaneously without a new deployment. ## Downsides There are a couple of downsides, as well: - Even with the above safety guards there is of course an increased risk from rolling out more often. As explained above though, this is also a benefit as we can more readily detect, root cause and address any regressions. - To be able to start using new Git features we would want to automatically bump the minimum required Git version to always match the version of the bundled fallback version of Git. This is a significant change to our current strategy and would potentially cause issues for downstream distributors of GitLab, like Linux distributions, which tend to have policies that they don't want to use bundled dependencies. We would have to remove the feature that allows those downstream users to use their own Git distribution. Given the upsides though these two downsides seem acceptable.