ci: Refactor fragile setup of unprivileged tests
GitLab Runner by default runs CI jobs as the root user. This is creating several problems for us when we want to test behaviour that relates to file permissions as the root user has special capabilities that allow it to just ignore those permissions altogether. To fix this, we run tests as an unprivileged user that lacks these capabilities.
The way this works is that we clone and build the code as root and then run the tests as unprivileged user. This both fixes above issues while also making sure that our tests never write into Gitaly's source tree directly, which we used to do some time ago.
This process is really fragile though: while we're reusing the Go mod and build cache, these caches have been populated by the root user and are thus only readable by us. So if the tests need to write anything to those caches then they will fail because the unprivileged user lacks the permission to do so. And while this has somehow worked until now, this does break with Go 1.19. We could try to do introduce more magic here, or make the caches writeable. But this only adds more workarounds on top of the already-complicated build process, so this doesn't feel like the right thing to do.
Instead, refactor our CI job to both build and test as the unprivileged
user. While sources are still owned by the root user, we manually create
the _build
directory with the unprivileged user as its owner and adapt
a few variables so that all build artifacts are created inside of that
directory. This ensures compatibility with Go 1.19 as we don't rely on
any fragile caching logic in Go anymore and retains our ability to run
tests without any special permissions.