refs: implement ref format migration

this patch series implements migration of ref storage formats such that it is for example possible to migrate a repository with "files" format to the "reftable" format. The migration logic comes in the form of a new git refs migrate subcommand. As mentioned in 1, I do have plans to extend the new git-refs(1) over time.

In the current form, the migration logic has three major limitations:

• It does not work with repositories that have worktrees as we have to migrate multiple ref stores here, one for every worktree. I wanted to avoid making this series too complex right from the start.

• It does not migrate reflogs, because we have no interfaces to do so. I want to eventually address this by adding log-only updates to transactions.

• It is not safe with concurrent writers. This is the limitation that is most critical in my eyes. The root cause here is that it is inherently impossible to lock the "files" backend for writes. I have been thinking about this issue a lot and have not found any solution that works. There are partial solutions here:

  • Create a central lockfile for the "files" backend -- if there, the backend will refuse to write. If that lock needs to be acquired during the "commit" phase of transaction though then we would essentially start to sequentialize all writes in the "files" backend, which is a non-starter. If not, then processes which are running already may not have seen it, and thus the issue still exists.

  • Pack all loose refs, remove "refs/" and create a "refs.lock" file. This isn't safe though because root refs can still be written.

  • Create a log of concurrent writes and apply that to the migrated refs once done. This is a lot of complexity, and it's unclear whether it even solves the issue with already-running writers.

  • Create a temporary "extensions.refMigration" extension that is unhandled by Git. New processes will refuse to run in such a repo and thus cannot write to it. Again, unsafe with running writers.

  Another alternative is that we could just make this a best effort. The "reftable" backend supports locking, and for the "files" backend we could just lock "HEAD" and call it a day. I'm not sure whether it is preferable though to have a "partial" solution compared to having none at all, as it may cause users to be less mindful. That's why I decided to just have no solution at all and document the limitation accordingly.

  If anybody has ideas here I'd be very happy to hear them.

Anyway. The current state of this patch series is sufficient to migrate repos without reflogs and worktrees, and thus mostly applies to bare repositories, only. This is somewhat intentional -- as a server operator this is of course our primary usecase at GitLab. We do plan to also upstream support for writing reflogs though, but in a later step. We do not plan to implement support for migrating repositories with worktrees, but I'd be happy to help out with the effort in case somebody else wants to.

The series is built on top of 4365c6fc (The sixth batch, 2024-05-20). It pulls in the following two dependencies:

• ps/refs-without-the-repository-updates at 00892786 (refs/packed: remove references to the_hash_algo, 2024-05-17). This is mostly to avoid conflicts.

• ps/pseudo-ref-terminology at 8e4f5c2d (refs: refuse to write pseudorefs, 2024-05-15). This is a functional prerequisite because the migration logic relies on the new definition of pseudorefs.

There are two minor textual conflicts when merged to "next" or "seen". One is a conflicting added header in "refs/reftable-backend.c", and one is a conflict with added functions in "refs.c". Both of these conflicts are trivial to solve by accepting both sides.

Closes Minimum viable candidate for migrating ref formats (#220 - closed). Part of Introduce a new command to work with refs (#317 - closed).