Skip to content

Add an RPC to apply a BFG object map to a repository

Nick Thomas requested to merge (removed):bfg-cleaner-rpc into master

In https://gitlab.com/gitlab-org/gitlab-ce/issues/19376#note_118309970 , we're enabling customers to reduce the size of their repositories without administrator intervention by recommending they:

  • Run the BFG Repo Cleaner against their repository
  • Force-push the cleaned repository to GitLab
  • Upload the generated object map to GitLab, allowing us to remove (or rewrite) any internal references that still point to commits rewritten by the BFG run

By internal references, I mean:

  • refs/environments/*
  • refs/keep-around/*
  • refs/merge-requests/*

Without removing those references and performing housekeeping, many objects removed by the BFG will stay in the repository indefinitely.

Removing those objects may cause various parts of GitLab to break. That's accepted, and something we're committed to fixing as the items come up.

This RPC is incredibly special-purpose. I've named it Apply, since the first implementation will simply delete the refs. A future iteration may attempt to rewrite them instead.

I've named BFG directly since the object maps we're passing are generated by it. That object map is simply a text file with two SHAs per line: an old commit SHA, followed by a new commit SHA - but it's a BFG-specific thing. Other tools generate other things.

I did consider an approach that would give us an RPC we could transition into "Run the BFG for me" over time, but since BFG is Java, that iteration is probably a very long way away, if ever.

Merge request reports