Windows Docker Runner errors since 17.5.0

Summary

Docker for Windows, using windows containers fails with errors for 17.5.0 and 17.5.1

For example:

Preparing environment
00:01
ERROR: Container "94d8ba0b0707915c0c042e2c8496067722016af5036631f4f629381d2c6e91c4" not found or removed. Will retry...
ERROR: Job failed (system failure): prepare environment: Error response from daemon: network bridge not found (exec.go:77:0s). Check https://docs.gitlab.com/runner/shells/index.html#shell-profile-loading for more information

Steps to reproduce

Use a simple .Net Framework application and a windows container image. Eg image: mcr.microsoft.com/dotnet/framework/sdk:4.8
Update the gitlab-runner exe from 17.4.1 to 17.5.0/17.5.1
Try to run the job

.gitlab-ci.yml

Build TestApp:
  variables:
    ARTIFACTS_NAME: TestApp
    PUBLISH_PROJECT: "./TestApp/TestApp.csproj"
  stage: Build
  before_script:
  - dotnet nuget update source testdomain.io-internal -u $PROGET_USER -p $PROGET_PASSWORD
    --valid-authentication-types "basic" --configfile "nuget.config" --store-password-in-clear-text
  - dotnet nuget update source testdomain.io -u $PROGET_USER -p $PROGET_PASSWORD
    --valid-authentication-types "basic" --configfile "nuget.config" --store-password-in-clear-text
  - dotnet --info
  image: mcr.microsoft.com/dotnet/framework/sdk:4.8
  needs: []
  tags:
  - windows
  script:
  - dotnet publish --nologo -c Release -r win-x64 /p:Platform=AnyCPU $PUBLISH_PROJECT
    --self-contained true -v q -o artifacts/$ARTIFACTS_NAME
  - "$Version = (Get-Item ./artifacts/$ARTIFACTS_NAME/$ARTIFACTS_NAME.exe).VersionInfo.ProductVersion"
  - Compress-Archive -Path "./artifacts/$ARTIFACTS_NAME/*" -DestinationPath "./artifacts/$ARTIFACTS_NAME-$Version.zip"
  artifacts:
    when: always
    paths:
    - artifacts/$ARTIFACTS_NAME*.zip
    expire_in: 1 week

Actual behavior

Job errors out

Expected behavior

Expect the job to build without errors

Relevant logs and/or screenshots

job log

Running with gitlab-runner 17.5.0 (66a723c3)
  on prd-win-grun-04 Yhwvqm67Z, system ID: s_72b3bba88b35
Resolving secrets
Preparing the "docker-windows" executor
01:02
Using Docker executor with image mcr.microsoft.com/dotnet/framework/sdk:4.8 ...
Using helper image:  registry.gitlab.com/gitlab-org/gitlab-runner/gitlab-runner-helper:x86_64-v17.5.0-servercore21H2
Pulling docker image registry.gitlab.com/gitlab-org/gitlab-runner/gitlab-runner-helper:x86_64-v17.5.0-servercore21H2 ...
Using docker image sha256:321b24590d76434b47ad235ce0f464b805ab8a730156a5a400ffdfe9be997757 for registry.gitlab.com/gitlab-org/gitlab-runner/gitlab-runner-helper:x86_64-v17.5.0-servercore21H2 with digest registry.gitlab.com/gitlab-org/gitlab-runner/gitlab-runner-helper@sha256:cc7f0887278efc6290a1e3e8d43ff798589029396237895e7952a59ac8e39993 ...
Using locally found image version due to "if-not-present" pull policy
Using docker image sha256:5dab0bd985d90e5637878b8f972ee0e229f3fa9f790299e82895a5b407d34bca for mcr.microsoft.com/dotnet/framework/sdk:4.8 with digest mcr.microsoft.com/dotnet/framework/sdk@sha256:96f6d277d5d574fce9fc13d5edc110db7809ab11c1bbace1be2f3bcf5e0032e2 ...
Preparing environment
00:01
ERROR: Container "94d8ba0b0707915c0c042e2c8496067722016af5036631f4f629381d2c6e91c4" not found or removed. Will retry...
ERROR: Job failed (system failure): prepare environment: Error response from daemon: network bridge not found (exec.go:77:0s). Check https://docs.gitlab.com/runner/shells/index.html#shell-profile-loading for more information

working job log (partial)


\[0KRunning with gitlab-runner 17.4.1 (32fe5502)\[0;m \[0K on prd-win-grun-04 Yhwvqm67Z, system ID: s_72b3bba88b35\[0;m \[0K\[36;1mResolving secrets\[0;m\[0;m section_start:1729568719:prepare_executor \[0K\[0K\[36;1mPreparing the "docker-windows" executor\[0;m\[0;m \[0KUsing Docker executor with image mcr.microsoft.com/dotnet/framework/sdk:4.8 ...\[0;m \[0KUsing helper image: registry.gitlab.com/gitlab-org/gitlab-runner/gitlab-runner-helper:x86_64-v17.4.1-servercore21H2\[0;m \[0KPulling docker image registry.gitlab.com/gitlab-org/gitlab-runner/gitlab-runner-helper:x86_64-v17.4.1-servercore21H2 ...\[0;m \[0KUsing docker image sha256:852110c45952a0921adb69a60f7524fad55cf9a63eba2ffe2b0b13c2eecbbf16 for registry.gitlab.com/gitlab-org/gitlab-runner/gitlab-runner-helper:x86_64-v17.4.1-servercore21H2 with digest registry.gitlab.com/gitlab-org/gitlab-runner/gitlab-runner-helper@sha256:a875c1bcde229a19052d866aa40ce17726079f3f6e10971ab416779fbca920ed ...\[0;m \[0KUsing locally found image version due to "if-not-present" pull policy\[0;m \[0KUsing docker image sha256:5dab0bd985d90e5637878b8f972ee0e229f3fa9f790299e82895a5b407d34bca for mcr.microsoft.com/dotnet/framework/sdk:4.8 with digest mcr.microsoft.com/dotnet/framework/sdk@sha256:96f6d277d5d574fce9fc13d5edc110db7809ab11c1bbace1be2f3bcf5e0032e2 ...\[0;m section_end:1729568782:prepare_executor \[0Ksection_start:1729568782:prepare_script \[0K\[0K\[36;1mPreparing environment\[0;m\[0;m Running on RUNNER-YHWVQM67 via prd-win-grun-04... section_end:1729568789:prepare_script \[0Ksection_start:1729568789:get_sources \[0K\[0K\[36;1mGetting source from Git repository\[0;m\[0;m \[32;1mFetching changes with git depth set to 20...\[0;m Initialized empty Git repository in C:/builds/REDACTED/.git/ \[32;1mCreated fresh repository.\[0;m \[32;1mChecking out 61fe628c as detached HEAD (ref is feature/DEVOPS-1379-runner-issue)...\[0;m git-lfs/3.5.1 (GitHub; windows amd64; go 1.21.7; git e237bb3a)

\[32;1mUpdating/initializing submodules recursively with git depth set to 20...\[0;m \[32;1mUpdated submodules\[0;m git-lfs/3.5.1 (GitHub; windows amd64; go 1.21.7; git e237bb3a) section_end:1729568815:get_sources \[0Ksection_start:1729568815:step_script \[0K\[0K\[36;1mExecuting "step_script" stage of the job script\[0;m\[0;m \[0KUsing docker image sha256:5dab0bd985d90e5637878b8f972ee0e229f3fa9f790299e82895a5b407d34bca for mcr.microsoft.com/dotnet/framework/sdk:4.8 with digest mcr.microsoft.com/dotnet/framework/sdk@sha256:96f6d277d5d574fce9fc13d5edc110db7809ab11c1bbace1be2f3bcf5e0032e2 ...\[0;m \[32;1m$ dotnet nuget update source REDACTED.io-internal -u $PROGET_USER -p <span dir="">PROGET_PASSWORD --valid-authentication-types "basic" --configfile "nuget.config" --store-password-in-clear-text\[0;m Package source "REDACTED.io-internal" was successfully updated. \[32;1m</span> dotnet nuget update source REDACTED.io -u $PROGET_USER -p <span dir="">PROGET_PASSWORD --valid-authentication-types "basic" --configfile "nuget.config" --store-password-in-clear-text\[0;m Package source "REDACTED.io" was successfully updated. \[32;1m</span> dotnet --info\[0;m

Environment description

We are using a custom install for a standalone GitLab instance.

The runners in question are using gitlab-runner-windows-amd64.exe on Windows Servers 2022 (21H2) using Docker version 27.2.0, build 3ab4256

We're using the docker-windows runner.

config.toml contents

concurrent = 2
check_interval = 0
shutdown_timeout = 0

[session_server]
  session_timeout = 1800

[[runners]]
  name = "prd-win-grun-04"
  url = "https://gitlab.REDACTED"
  id = 138
  token = "REDACTED"
  token_obtained_at = 2024-09-06T01:03:50Z
  token_expires_at = 0001-01-01T00:00:00Z
  executor = "docker-windows"
  shell = "powershell"
  [runners.custom_build_dir]
  [runners.cache]
    Type = "s3"
    Path = "windows"
    Shared = true
    [runners.cache.s3]
      ServerAddress = "s3.amazonaws.com"
      AccessKey = "REDACTED"
      SecretKey = "REDACTED"
      BucketName = "REDACTED-cache"
      BucketLocation = "REDACTED"
      Insecure = false
      ServerSideEncryption = "KMS"
      ServerSideEncryptionKeyID = "alias/aws/s3"
  [runners.docker]
    tls_verify = false
    image = "mcr.microsoft.com/dotnet/sdk:6.0-windowsservercore-ltsc2022"
    cpus = "2"
    privileged = false
    disable_entrypoint_overwrite = false
    oom_kill_disable = false
    disable_cache = true
    shm_size = 0
    pull_policy = ["if-not-present"]

Used GitLab Runner version

Version:      17.5.0
Git revision: 66a723c3
Git branch:   17-5-stable
GO version:   go1.22.7
Built:        2024-10-17T15:26:50+0000
OS/Arch:      windows/amd64

Possible fixes

Workarounds as of 2024-10-25

Upgrade to GitLab Runner 17.5.2
Set the FF_NETWORK_PER_BUILD feature flag.
Set a network_mode in the runner config.

Edited Oct 25, 2024 by Darren Eastman