Skip to content

Populate remote branches in-memory via `ls-remote` rather than using `fetch`

Confidential due to being related to gitlab#38386 (closed).

Currently when pushing to a remote mirror, gitlab-rails fetches the remote, which causes all of the objects on the remote to be available in the on-disk repository of the project being pushed.

As a result, the "Canonical to Security" push mirroring we've set up leaks security commits. If someone knows the commit SHA of a security fix, they can view it in the Canonical repository even though the commit should only exist in the Security project prior to publication. One easy way to know a security commit SHA is viewing the help page after we've deployed.

In order to avoid exposing security fixes before they're intended to be public, we need to avoid fetching a remote mirror, or at least avoid persisting that fetch on disk.

One proposal from @jacobvosmaer-gitlab was to instead utilize ls-remote and parse the output, so that the objects on the remote are only ever stored in memory. He mentioned today that @samihiltunen noticed we already do this when listing remote tags.

A rough idea of a proper fix for this may be:

  1. In gitlab-rails, avoid fetching the remote mirror prior to performing an UpdateRemoteMirrorRequest RPC.
  2. In Gitaly, don't assume that Rugged has references to remote objects in the local repository.
  3. Use ls-remote --heads to enumerate branches, parse the output, and provide an array of Gitlab::Git::Branch objects.
  4. Put this all behind feature flags so that we can quickly revert to the existing behavior.

Timeline and plan for rollout

  1. !2086 (merged) adds an experiment to fetch branches via ls-remote and compares it to existing functionality.
  2. Enable the experiment feature flag. => https://gitlab.com/gitlab-com/gl-infra/feature-flag-log/-/issues/1044
  3. Deploy a Gitaly version no earlier than 1bd66a4c to production. => PENDING
  4. Run the above experiment for at least one week in production. Search production logs for experimental_remote_branches returned differing values from control.
  5. Deploy a Gitaly version no earlier than 8122214a to production.
  6. Enable feature flag: /chatops run feature set gitaly_ruby_remote_branches_ls_remote true
  7. Verify SSH mirroring still works.
  8. Switch to using ls-remote exclusively, behind a feature flag. => !2183 (merged) / gitlab!30877 (merged)
  9. Keep the experimental behavior as the exclusive functionality for at least one week in production. First deploy: Deployed June 4, feature flag enabled https://gitlab.com/gitlab-com/gl-infra/feature-flag-log/-/issues/1299.
  10. Invert the flag's behavior -- old functionality behind a feature flag, new behavior by default. We might want to ship a self-managed release with this behavior. => !2330 (merged) / gitlab!35781 (merged)
  11. Remove the feature flag, including the fetch from gitlab-rails. => !2417 (merged) / gitlab!37760 (merged)
Edited by Robert Speicher
To upload designs, you'll need to enable LFS and have an admin enable hashed storage. More information