git ls-remote intermittently returns zero branches

Possible solution

Ensure lookup-cache=positive is set in the NFS client mount options and review our other recommended NFS mount options.

Summary

Large GitLab Premium Customer raised a ticket (internal links) as some of their projects' Jenkins pipelines were not running correctly. Intermittently, Jenkins would run pipelines for all branches in the projects, causing significant load on this infrastructure.

• Jenkins' git plugin polls git repos using git ls-remote -h -- <https url>.
• Customer is running Gitaly across three load balanced servers, all sharing NFS storage.
• Investigation reproduced this behaviour using git ls-remote running in a simple loop of 3-5 seconds.
• Deletion of branches identified as a potential trigger for this misbehaviour.

Observations

• The following git parameter for ls-remote might result in this issue being handled better on the client.

--exit-code
    Exit with status "2" when no matching refs are found in the remote repository.
    Usually the command exits with status "0" to indicate it successfully talked
    with the remote repository, whether it found any matching refs.   

Steps to reproduce

• Script written to poll repository(s) and log zero branches being returned.
  • When this issue was raised, the script was tested with ssh git remotes. Reviewing the Jenkins log output (attached to an earlier ticket - internal link) indicates https is being used. It reproduces with either.
• While polling (create and) delete branches from the repository.
• We suspect that NFS is a contributing factor.

Example Project

(If possible, please create an example project here on GitLab.com that exhibits the problematic behavior, and link to it here in the bug report)

(If you are using an older version of GitLab, this will also determine whether the bug is fixed in a more recent version)

What is the current bug behavior?

Polling a GitLab hosted git repository with git ls-remote returns zero branches.

What is the expected correct behavior?

More than zero branches returned.

Relevant logs and/or screenshots

(Paste any relevant logs - please use code blocks (```) to format console output, logs, and code as it's tough to read otherwise.)

Output of checks

(If you are reporting a bug on GitLab.com, write: This bug happens on GitLab.com)

Results of GitLab environment info

customer is running GitLab 12.10.

Possible fixes

(If you can, link to the line of code that might be responsible for the problem)