Skip to content

Git shallow clone strategy omits (remote) branches

Everyone can contribute. Help move this issue forward while earning points, leveling up and collecting rewards.

Summary

When doing shallow clones (DEPTH=1+), not all remote branches are retrieved, as a matter of fact, only 'reachable' or relevant branches are fetched. The default in gitlab is 50, but even this does not guarantee all branches are available.

Steps to reproduce

In our project, we want to create architecture specific branches from our main release branch. What the CI will do, checkout the current branch (gitlab-runner does this of course by default), and go over a list of available existing architecture branches to update these. To do so, it uses git branch --list --remote "origin/release/v0/*" | sed -n -r 's|.*origin/release/v0/||p' to obtain the list of 'v0' named branches. E.g. release/v0/arm32v7, release/v0/arm64v8.

However, if git fetch is set with a low depth parameter. We can create a scenario where not all remotes have been retrieved. This is related to the fact that the branches could not be reached in the current 'depth'.

The work around for now appears to be either add a git fetch to the CI task (ugly) or (according to the documentation) set the depth to 0.

.gitlab-ci.yml 
.git:
  <<: *common_parameters
  image: registry.hub.docker.com/gitscm/git:latest
  before_script:
    - git config --local user.name "${GITLAB_USER_NAME}"
    - git config --local user.email "${GITLAB_USER_EMAIL}"
    - git config --local credential.helper "cache --timeout=2147483647"
    - printf "url=${CI_PROJECT_URL}\nusername=esbs-bot\npassword=${CI_PERSONAL_TOKEN}\n\n" | git credential approve
    - git remote set-url --push origin "https://esbs-bot@${CI_REPOSITORY_URL#*@}"
  after_script:
    - git credential-cache exit

create_release_branches:
  extends: .git
  stage: release
  only:
    refs:
      - /^v\d+\.\d+$/
  except:
    refs:
      - branches # Workaround for gitlab-org/gitlab-ce#27818
  script:
    - git_tag="$(git describe --exact-match --match "v[0-9]*")"
    - git_tag_msg="$(git tag --format="%(contents:subject)%0a%0a%(contents:body)%0a%0a(Auto-created release candidate)" --list "${git_tag}")"
    - |
      git fetch --prune # Added as workaround
      for release_branch in "release" $(git branch --list --remote "origin/stable/v0/*" | sed -n -r 's|.*origin/stable/v0/||p'); do
        echo "Creating release branch '${release_branch}' from '${CI_COMMIT_REF_NAME}' (${CI_COMMIT_SHORT_SHA})"
        git checkout "${CI_COMMIT_SHA}" -b "stable/${CI_COMMIT_TAG:?}/${release_branch}"
        if [ "${release_branch}" != "release" ]; then
          sed -i 's|FROM registry.hub.docker.com/library/alpine|FROM registry.hub.docker.com/'"${release_branch}"'/alpine|g' "Dockerfile"
          git add "Dockerfile"
          git commit --message="${release_branch}: Auto-created release branch 'stable/${CI_COMMIT_TAG:?}/${release_branch}'." --no-edit
          release_prefix="${release_branch}/"
        fi
        git tag --annotate --message="${git_tag_msg:-See tag '${CI_COMMIT_TAG}'}" "${release_prefix:-}${CI_COMMIT_TAG}.0-rc1"
        git push --follow-tags origin "HEAD"
      done

Actual behavior

All remotes are available on the runner, albeit at a shallow depth of the main branch

Expected behavior

Even shallow fetch, should always retrieve all branches, not just the current branch.

Proposal

As stated below in #30067 (comment 215829463) not fetching all the branches is expected since we are only going far back X commits.

However, git clone --depth supports an additional argument --no-single-branch

--depth

Create a shallow clone with a history truncated to the specified number of commits. Implies --single-branch unless --no-single-branch is given to fetch the histories near the tips of all branches. If you want to clone submodules shallowly, also pass --shallow-submodules. which should resolve this issue, it clones all branches (at depth=).

The proposal is thus by default, enable this flag, so we get expected behavior in that all expected branches are available.

If processing is to heavy or this is too big a bandwith burden, this could be added as GIT_SINGLE_BRANCH="yes|no".

Edited by 🤖 GitLab Bot 🤖