Fix windows nanoserver by replacing sh.exe with busybox

Guillaume Chauvelrequested to merge
gitlab-community/gitlab-org/ci-cd/runner-tools/base-images:gchauvel/fix-runner-helper-nanoserver into main

What does this MR do?

This MR is inspired by !43 (closed)

It allows gitlab-org/gitlab-runner#39333 by:

  • replacing MinGit with MinGit busybox for windows nanoserver amd64 images.
  • replacing MinGit's sh.exe with busyboxa64.exe for windows nanoserver arm64 images, as an official MinGit busybox doesn't exist for arm64

⚠️ In the "common" files, there is currently information for using a personal repository with a deploy token that is permanent and read-only. We should implement something equivalent using a project owned by GitLab, such as: https://gitlab.com/gitlab-org/ci-cd/gitlab-runner-pipeline-tests/submodules/mixed-submodules-test⚠️

the container-tests scripts:

  • clone-*-helper-env-var
  • clone-*-insteadof
  • clone-*-url

It tests the 2 ways gitlab-runner uses to retrieve git repositories, as well as a previous one (see below).

In order to run test, properly tagged runners to run pre-configured Windows (docker engine, container/hyper-v feature, etc...) are required.

Why was this MR needed?

There are issues with nanoserver ltsc2022 and ltsc2025

I had analyzed nanoserver behavior which was summarized in gitlab-runner#38844. However, the approach described in work item 38844, note 2738628015 ("git cred helper is implemented in the runner-helper binary") — and which can be found at !43 (closed) cannot work. I had shown in the description of gitlab-runner!5447, sh is always invoked to run a credential helper, even when the helper is a pure binary not a script. A shshell is therefore required on Linux and Windows regardless of whether the helper itself is a standalone executable.

nanoserver test

I have updated my test repository that reproduces how GitLab Runner fetches repositories: https://gitlab.com/guillaume.chauvel/testpwshgitcredhelper
The CI results are available here: https://gitlab.com/guillaume.chauvel/testpwshgitcredhelper/-/jobs/13785037916
The script tests repository retrieval on nanoserver:ltsc2019, ltsc2022, and ltsc2025 matching GitLab Runner behavior with various parameters:

  • Credential methods:
    • FF_GIT_URLS_WITHOUT_TOKENS ON: Credential Helper
    • FF_GIT_URLS_WITHOUT_TOKENS OFF — Credential in external file (after !5912)
    • FF_GIT_URLS_WITHOUT_TOKENS OFF — Password in URI (before !5912)
  • Git distributions:
    • PortableGit
    • MinGit
    • MinGit-BusyBox
  • Submodule
  • bonus: repository format sha256
  • bonus: LFS

Results on custom nanoserver images (3 git distributions)

Method Operation nanoserver:ltsc2019 nanoserver:ltsc2022 nanoserver:ltsc2025
FF_GIT_URLS_WITHOUT_TOKENS ON fetch ✅ all 3 ❌ PortableGit & MinGit FAIL / ✅ MinGit-BusyBox ❌ PortableGit & MinGit FAIL / ✅ MinGit-BusyBox
FF_GIT_URLS_WITHOUT_TOKENS ON submodule ✅ all 3 ❌ PortableGit & MinGit FAIL / ✅ MinGit-BusyBox ❌ PortableGit & MinGit FAIL / ✅ MinGit-BusyBox
FF_GIT_URLS_WITHOUT_TOKENS OFF — External Credential fetch ✅ all 3 ✅ all 3 ✅ all 3
FF_GIT_URLS_WITHOUT_TOKENS OFF — External Credential submodule ✅ all 3 ❌ PortableGit & MinGit FAIL / ✅ MinGit-BusyBox ❌ PortableGit & MinGit FAIL / ✅ MinGit-BusyBox
FF_GIT_URLS_WITHOUT_TOKENS OFF — Password in URI fetch ✅ all 3 ✅ all 3 ✅ all 3

Results on actual runner-helper nanoserver base images

The runner-helper base images ship MinGit (not MinGit-BusyBox), so they hit the failures:

Method Operation ltsc2019 ltsc2022 ltsc2025
FF_GIT_URLS_WITHOUT_TOKENS ON fetch and/or submodule ✅ ❌ FAIL ❌ FAIL
FF_GIT_URLS_WITHOUT_TOKENS OFF — External Credential fetch ✅ ✅ ✅
FF_GIT_URLS_WITHOUT_TOKENS OFF — External Credential submodule ✅ ❌ FAIL ❌ FAIL
FF_GIT_URLS_WITHOUT_TOKENS OFF — Password in URI fetch ✅ ✅ ✅

Root cause

Nanoserver ltsc2022 and ltsc2025 are missing system DLLs (or symbols) required by GNU bash (sh.exe). Both the credential helper mechanism and submodule operations invoke sh.exe internally. Only MinGit-BusyBox is unaffected because it ships its own statically-linked sh.exe.

This means that on nanoserver ltsc2022/ltsc2025 with PortableGit/MinGit:

  • Simple git fetch works (with External Credential or Password in URI)
  • But FF_GIT_URLS_WITHOUT_TOKENS ON is entirely broken
  • And submodules are broken regardless of credential method

Using MinGit-BusyBox instead of PortableGit/MinGit in the nanoserver helper images would resolve all these failures, as it ships its own statically-linked sh.exe that does not depend on the missing system DLLs.

Related Issues, MRs

gitlab-org/gitlab-runner#38814 (closed)
gitlab-org/gitlab-runner#38844 (closed)
gitlab-org/gitlab-runner#39333
gitlab-org/gitlab-runner#39418
gitlab-org/gitlab-runner!5525 (merged)

Edited by Guillaume Chauvel

Merge request reports