Populate remote branches in-memory via `ls-remote` rather than using `fetch`
Currently when pushing to a remote mirror, gitlab-rails fetches the remote, which causes all of the objects on the remote to be available in the on-disk repository of the project being pushed.
As a result, the "Canonical to Security" push mirroring we've set up leaks security commits. If someone knows the commit SHA of a security fix, they can view it in the Canonical repository even though the commit should only exist in the Security project prior to publication. One easy way to know a security commit SHA is viewing the help page after we've deployed.
In order to avoid exposing security fixes before they're intended to be public, we need to avoid fetching a remote mirror, or at least avoid persisting that fetch on disk.
One proposal from @jacobvosmaer-gitlab was to instead utilize ls-remote
and parse the output, so that the objects on the remote are only ever stored in memory. He mentioned today that @samihiltunen noticed we already do this when listing remote tags.
A rough idea of a proper fix for this may be:
- In gitlab-rails, avoid fetching the remote mirror prior to performing an
UpdateRemoteMirrorRequest
RPC. - In Gitaly, don't assume that Rugged has references to remote objects in the local repository.
- Use
ls-remote --heads
to enumerate branches, parse the output, and provide an array ofGitlab::Git::Branch
objects. - Put this all behind feature flags so that we can quickly revert to the existing behavior.
Timeline and plan for rollout
-
!2086 (merged) adds an experiment to fetch branches via ls-remote
and compares it to existing functionality. -
Enable the experiment feature flag. => https://gitlab.com/gitlab-com/gl-infra/feature-flag-log/-/issues/1044 -
Deploy a Gitaly version no earlier than 1bd66a4c to production. => PENDING -
Run the above experiment for at least one week in production. Search production logs for experimental_remote_branches returned differing values from control
. -
Deploy a Gitaly version no earlier than 8122214a to production. -
Enable feature flag: /chatops run feature set gitaly_ruby_remote_branches_ls_remote true
-
Verify SSH mirroring still works. -
Switch to using ls-remote
exclusively, behind a feature flag. => !2183 (merged) / gitlab!30877 (merged) -
Keep the experimental behavior as the exclusive functionality for at least one week in production. First deploy: Deployed June 4, feature flag enabled https://gitlab.com/gitlab-com/gl-infra/feature-flag-log/-/issues/1299. -
Invert the flag's behavior -- old functionality behind a feature flag, new behavior by default. We might want to ship a self-managed release with this behavior. => !2330 (merged) / gitlab!35781 (merged) -
Remove the feature flag, including the fetch
fromgitlab-rails
. => !2417 (merged) / gitlab!37760 (merged)