Skip to content

GitLab Next

  • Menu
Projects Groups Snippets
  • Help
    • Help
    • Support
    • Community forum
    • Submit feedback
    • Contribute to GitLab
  • Sign in / Register
  • gitaly gitaly
  • Project information
    • Project information
    • Activity
    • Labels
    • Members
  • Repository
    • Repository
    • Files
    • Commits
    • Branches
    • Tags
    • Contributors
    • Graph
    • Compare
    • Locked Files
  • Issues 556
    • Issues 556
    • List
    • Boards
    • Service Desk
    • Milestones
    • Iterations
    • Requirements
  • Merge requests 52
    • Merge requests 52
  • CI/CD
    • CI/CD
    • Pipelines
    • Jobs
    • Schedules
    • Test Cases
  • Deployments
    • Deployments
    • Environments
    • Releases
  • Packages & Registries
    • Packages & Registries
    • Container Registry
  • Monitor
    • Monitor
    • Incidents
  • Analytics
    • Analytics
    • Value stream
    • CI/CD
    • Code review
    • Insights
    • Issue
    • Repository
  • Snippets
    • Snippets
  • Activity
  • Graph
  • Create a new issue
  • Jobs
  • Commits
  • Issue Boards
Collapse sidebar

GitLab 15.0 is launching on May 22! This version brings many exciting improvements, but also removes deprecated features and introduces breaking changes that may impact your workflow. To see what is being deprecated and removed, please visit Breaking changes in 15.0 and Deprecations.

  • GitLab.org
  • gitalygitaly
  • Issues
  • #3083
Closed
Open
Created Aug 24, 2020 by Ben Prescott @bprescott_↙ ☺@bprescott_💡Developer

git ls-remote intermittently returns zero branches

Possible solution

Ensure lookup-cache=positive is set in the NFS client mount options and review our other recommended NFS mount options.

Summary

Large GitLab Premium Customer raised a ticket (internal links) as some of their projects' Jenkins pipelines were not running correctly. Intermittently, Jenkins would run pipelines for all branches in the projects, causing significant load on this infrastructure.

  • Jenkins' git plugin polls git repos using git ls-remote -h -- <https url>.
  • Customer is running Gitaly across three load balanced servers, all sharing NFS storage.
  • Investigation reproduced this behaviour using git ls-remote running in a simple loop of 3-5 seconds.
  • Deletion of branches identified as a potential trigger for this misbehaviour.

Observations

  • The following git parameter for ls-remote might result in this issue being handled better on the client.
--exit-code
    Exit with status "2" when no matching refs are found in the remote repository.
    Usually the command exits with status "0" to indicate it successfully talked
    with the remote repository, whether it found any matching refs.   

Steps to reproduce

  • Script written to poll repository(s) and log zero branches being returned.
    • When this issue was raised, the script was tested with ssh git remotes. Reviewing the Jenkins log output (attached to an earlier ticket - internal link) indicates https is being used. It reproduces with either.
  • While polling (create and) delete branches from the repository.
  • We suspect that NFS is a contributing factor.

Example Project

(If possible, please create an example project here on GitLab.com that exhibits the problematic behavior, and link to it here in the bug report)

(If you are using an older version of GitLab, this will also determine whether the bug is fixed in a more recent version)

What is the current bug behavior?

Polling a GitLab hosted git repository with git ls-remote returns zero branches.

What is the expected correct behavior?

More than zero branches returned.

Relevant logs and/or screenshots

(Paste any relevant logs - please use code blocks (```) to format console output, logs, and code as it's tough to read otherwise.)

Output of checks

(If you are reporting a bug on GitLab.com, write: This bug happens on GitLab.com)

Results of GitLab environment info

customer is running GitLab 12.10.

Possible fixes

(If you can, link to the line of code that might be responsible for the problem)

Edited Mar 01, 2021 by Ben Prescott @bprescott_↙ ☺
To upload designs, you'll need to enable LFS and have an admin enable hashed storage. More information
Assignee
Assign to
Time tracking