Properly cache gitaly and git binaries in CI

Issue

The real issue is that we weren't archiving the git binaries properly: !72839 (merged)

Here is a list of files cached for Gitaly:

updated-cache.txt

Initial research (outdated)

When Gitaly started requiring Git v2.33, but our system images used Git v2.31, we ran into a bit of a chicken and egg problem as seen in https://gitlab.com/gitlab-org/gitlab/-/jobs/1704115864#L65. setup-test-env fails because gitaly-test-build doesn't build Gitaly because the dir is restored from cache, but we need it to recompile git so it has the right version.

Since git wasn't rebuilt, setup-test-env fails in master because Gitaly wants Git v2.33, not v2.31.

I attempted to force a rebuild in !72823 (closed) if the git binary were not present. However, when the setup-test-env cache for Gitaly binaries is restored, it restores a subset of files in tmp/tests/gitaly. However, this tmp/tests/gitaly doesn't actually have all the source files, like go.mod and go.sum, so any attempts to build will fail.

It seems that we're in a world of pain right now because we're mixing source and binaries in the cache, and it's not easy to disentangle this. Ideally we should call Gitaly's make install with a PREFIX, install the binaries to $PREFIX/bin, and then cache that directory.

For now:

1. We have to revert the Gitaly server bump: !72828 (merged)
2. We can work around this problem by bumping our build images to use Git v2.33: gitlab-build-images!442 (merged)