Support a way to refresh packed-refs before fetch
A customer+ (ZD: http://gitlab.zendesk.com/agent/tickets/103632, https://gitlab.com/gitlab-org/gitlab-ce/issues/51437) is running into a NFS v4.0 client bug that is present in all recent versions of the Linux kernel (https://marc.info/?l=linux-nfs&m=153816500525563&w=2) that causes clones and pushes to a repository to fail if a git process does the following:
- Renames or unlinks
packed-refs
- Keeps an open file handle to
packed-refs
.
For example, this happens while git gc
is running because it packs all loose references, renames packed-refs.new
to packed-refs
, and keeps packed-refs
open. The newly-overwritten packed-files shows up with a "Stale file handle" on an Isilon NFS server until either:
-
ls
is issued to clear the NFS dentry/inode cache - The
git gc
process terminates and closes thepacked-refs
file
The "Stale file handle" error doesn't happen on a standard Linux NFS server; instead, the contents of the file is stale until one of the two actions above are taken.
This problem may also be caused by more than git gc
, but we haven't yet been able to determine yet what else may be causing this.
This problem is causing immediate problems for the customer. We've recommended a few actions:
- Disable automatic
git gc
via the Housekeeping setting - Make
git gc
run only on a separate Sidekiq host
Note that this customer is on GitLab 10.8 and is holding off on upgrading until things stabilize.
We might be able to workaround this NFS client bug by adding support for some pre-fetch hook that runs ls
on the repository directory. However, right now Git doesn't support a pre-fetch hook. I was hoping there was a Git hook that would allow us to run a script before any attempt to fetch, but as far as I can tell there isn't one (https://mirrors.edge.kernel.org/pub/software/scm/git/docs/githooks.html). The closest I found was the packObjectsHook (https://git-scm.com/docs/git-config#git-config-uploadpackpackObjectsHook), but this doesn't help us since the failure happens earlier in the ref negotiation.