Skip to content

Make submodule `--remote` more resilient

Hannes Hörlrequested to merge
hhoerl/359825-fetch-sub-remotes into main

What does this MR do?

Makes updating submodules with --remote more resilient.

Why was this MR needed?

We have seen issues with pulling submodules:

  • fatal: Unable to find refs/remotes/origin/<submoduleBranch> revision in submodule path '<submoduleDir>'

    When remote heads are not available when upadting a submodule's revision, we explicitly need to pull those refs, so that the submodule can be updated.

  • fatal: Needed a single revision
    Unable to find current origin/master revision in submodule path '<submoduleDir>'

    This seems to stem from old git versions, which default to remote/main as the default branch, rather than getting the degfault branch from the remote. While there is not much we can do here, we need to doc that and also check in our tests for that.

What's the best way to test this MR?

To test with the most edge cases I can think of:

  • run on powershell, so we ensure the additional script we render does not break anything (eg. it's quoted correctly)
    • additionally could be tested on bash & pwsh, but I am most concerned with powershell / windows
  • use private repos all the way, to ensure auth works for all (submodule) operations
    • additionally running with FF_GIT_URLS_WITHOUT_TOKENS could be tested
  • this manual test on k8s ensures completely isolated/ephemeral setup
    • the additional tests in the pipeline test with other setups, e.g. on windows with a relatively old git version
    • additionally the helper image could be swapped out, to test with different versions of git

Setup the repos:

  • create a submodule project sub-1 which is private
    • create a branch default-branch and add some commits
    • make the default-branch the default branch
    • create an oprhan branch branch-1 with a couple of commits
    • create an oprhan branch branch-2 with a couple of commits
  • create a super project sub-main which is private
    • submodule in sub-1's branch-1 at branch-1, e.g. git submodule add -b branch-1 ../sub-1 branch-1
    • submodule in sub-1's branch-2 at branch-2, e.g. git submodule add -b branch-2 ../sub-1 branch-2
    • submodule in sub-1s default branch at default, e.g. git submodule add ../sub-1 default
  • for all submodules, checkout a revision that is not the tip of the respective branch e.g. git submodule foreach 'git reset HEAD~1'
  • ensure that the .gitmodules in the super project
    • has set branch branch-1 for the submodule branch-1
    • has set branch branch-2 for the submodule branch-2
    • has not set a branch for the submodule default
  • commit everything and push sub-main & sub-1 to GL
  • ensure the superproject's token is in the allow list of the submodule project
    • go to the submodule project's Settings > CI/CD > Job token permissions > CI/CD job token allowlist
    • and add the superproject to the allow list

Setup a cluster to run jobs on:

  • create a cluster with linux & windows nodes

    cluster create 
    gcloud container clusters create cl-win --enable-ip-alias --num-nodes=1 --location europe-west1-c
gcloud container node-pools create winnodes --cluster=cl-win --image-type=WINDOWS_LTSC_CONTAINERD --no-enable-autoupgrade --machine-type=n1-standard-2 --windows-os-version=ltsc2019 --location europe-west1-c --num-nodes=1

  • use a config like:

    config.toml 
    listen_address = ":9252"
concurrent = 4
check_interval = 0
connection_max_age = "15m0s"
shutdown_timeout = 0
[session_server]
  session_timeout = 1800
[[runners]]
  name = "gke-win-0"
  url = "https://gitlab.com/"
  id = 0
  token = "glrt-NopeNopeNope"
  token_obtained_at = 0001-01-01T00:00:00Z
  token_expires_at = 0001-01-01T00:00:00Z
  executor = "kubernetes"
  shell = "powershell"
  environment = ["FF_USE_POWERSHELL_PATH_RESOLVER=true"]
  [runners.kubernetes]
    poll_timeout = 1200
    pull_policy = "always"
  [runners.kubernetes.node_selector]
    "kubernetes.io/arch" = "amd64"
    "kubernetes.io/os" = "windows"
    "node.kubernetes.io/windows-build" = "10.0.17763"

Run the pipeline an check that the subnodules are checked out at their respective revs:

  • .gitlab-ci.yaml 
    subs:
  stage: test
  image: registry.gitlab.com/gitlab-org/gitlab-runner/gitlab-runner-helper:x86_64-latest-servercore1809
  hooks:
    pre_get_sources_script:
      - git version
  variables: 
    # CI_DEBUG_TRACE: 1
    GIT_SUBMODULE_STRATEGY: recursive
    GIT_SUBMODULE_UPDATE_FLAGS: "--remote"
  script:
    - git submodule foreach 'git rev-parse HEAD'

  • ensure the submodules are checked out at the tip of the respective branches from sub-1

What are the relevant issue numbers?

Edited by Hannes Hörl

Merge request reports