Windows Docker Runner errors since 17.5.0
Summary
Docker for Windows, using windows containers fails with errors for 17.5.0 and 17.5.1
For example:
Preparing environment
00:01
ERROR: Container "94d8ba0b0707915c0c042e2c8496067722016af5036631f4f629381d2c6e91c4" not found or removed. Will retry...
ERROR: Job failed (system failure): prepare environment: Error response from daemon: network bridge not found (exec.go:77:0s). Check https://docs.gitlab.com/runner/shells/index.html#shell-profile-loading for more information
Steps to reproduce
- Use a simple .Net Framework application and a windows container image. Eg
image: mcr.microsoft.com/dotnet/framework/sdk:4.8
- Update the gitlab-runner exe from 17.4.1 to 17.5.0/17.5.1
- Try to run the job
.gitlab-ci.yml
Build TestApp:
variables:
ARTIFACTS_NAME: TestApp
PUBLISH_PROJECT: "./TestApp/TestApp.csproj"
stage: Build
before_script:
- dotnet nuget update source testdomain.io-internal -u $PROGET_USER -p $PROGET_PASSWORD
--valid-authentication-types "basic" --configfile "nuget.config" --store-password-in-clear-text
- dotnet nuget update source testdomain.io -u $PROGET_USER -p $PROGET_PASSWORD
--valid-authentication-types "basic" --configfile "nuget.config" --store-password-in-clear-text
- dotnet --info
image: mcr.microsoft.com/dotnet/framework/sdk:4.8
needs: []
tags:
- windows
script:
- dotnet publish --nologo -c Release -r win-x64 /p:Platform=AnyCPU $PUBLISH_PROJECT
--self-contained true -v q -o artifacts/$ARTIFACTS_NAME
- "$Version = (Get-Item ./artifacts/$ARTIFACTS_NAME/$ARTIFACTS_NAME.exe).VersionInfo.ProductVersion"
- Compress-Archive -Path "./artifacts/$ARTIFACTS_NAME/*" -DestinationPath "./artifacts/$ARTIFACTS_NAME-$Version.zip"
artifacts:
when: always
paths:
- artifacts/$ARTIFACTS_NAME*.zip
expire_in: 1 week
Actual behavior
Job errors out
Expected behavior
Expect the job to build without errors
Relevant logs and/or screenshots
job log
Running with gitlab-runner 17.5.0 (66a723c3)
on prd-win-grun-04 Yhwvqm67Z, system ID: s_72b3bba88b35
Resolving secrets
Preparing the "docker-windows" executor
01:02
Using Docker executor with image mcr.microsoft.com/dotnet/framework/sdk:4.8 ...
Using helper image: registry.gitlab.com/gitlab-org/gitlab-runner/gitlab-runner-helper:x86_64-v17.5.0-servercore21H2
Pulling docker image registry.gitlab.com/gitlab-org/gitlab-runner/gitlab-runner-helper:x86_64-v17.5.0-servercore21H2 ...
Using docker image sha256:321b24590d76434b47ad235ce0f464b805ab8a730156a5a400ffdfe9be997757 for registry.gitlab.com/gitlab-org/gitlab-runner/gitlab-runner-helper:x86_64-v17.5.0-servercore21H2 with digest registry.gitlab.com/gitlab-org/gitlab-runner/gitlab-runner-helper@sha256:cc7f0887278efc6290a1e3e8d43ff798589029396237895e7952a59ac8e39993 ...
Using locally found image version due to "if-not-present" pull policy
Using docker image sha256:5dab0bd985d90e5637878b8f972ee0e229f3fa9f790299e82895a5b407d34bca for mcr.microsoft.com/dotnet/framework/sdk:4.8 with digest mcr.microsoft.com/dotnet/framework/sdk@sha256:96f6d277d5d574fce9fc13d5edc110db7809ab11c1bbace1be2f3bcf5e0032e2 ...
Preparing environment
00:01
ERROR: Container "94d8ba0b0707915c0c042e2c8496067722016af5036631f4f629381d2c6e91c4" not found or removed. Will retry...
ERROR: Job failed (system failure): prepare environment: Error response from daemon: network bridge not found (exec.go:77:0s). Check https://docs.gitlab.com/runner/shells/index.html#shell-profile-loading for more information
working job log (partial)
\[0KRunning with gitlab-runner 17.4.1 (32fe5502)\[0;m \[0K on prd-win-grun-04 Yhwvqm67Z, system ID: s_72b3bba88b35\[0;m \[0K\[36;1mResolving secrets\[0;m\[0;m section_start:1729568719:prepare_executor \[0K\[0K\[36;1mPreparing the "docker-windows" executor\[0;m\[0;m \[0KUsing Docker executor with image mcr.microsoft.com/dotnet/framework/sdk:4.8 ...\[0;m \[0KUsing helper image: registry.gitlab.com/gitlab-org/gitlab-runner/gitlab-runner-helper:x86_64-v17.4.1-servercore21H2\[0;m \[0KPulling docker image registry.gitlab.com/gitlab-org/gitlab-runner/gitlab-runner-helper:x86_64-v17.4.1-servercore21H2 ...\[0;m \[0KUsing docker image sha256:852110c45952a0921adb69a60f7524fad55cf9a63eba2ffe2b0b13c2eecbbf16 for registry.gitlab.com/gitlab-org/gitlab-runner/gitlab-runner-helper:x86_64-v17.4.1-servercore21H2 with digest registry.gitlab.com/gitlab-org/gitlab-runner/gitlab-runner-helper@sha256:a875c1bcde229a19052d866aa40ce17726079f3f6e10971ab416779fbca920ed ...\[0;m \[0KUsing locally found image version due to "if-not-present" pull policy\[0;m \[0KUsing docker image sha256:5dab0bd985d90e5637878b8f972ee0e229f3fa9f790299e82895a5b407d34bca for mcr.microsoft.com/dotnet/framework/sdk:4.8 with digest mcr.microsoft.com/dotnet/framework/sdk@sha256:96f6d277d5d574fce9fc13d5edc110db7809ab11c1bbace1be2f3bcf5e0032e2 ...\[0;m section_end:1729568782:prepare_executor \[0Ksection_start:1729568782:prepare_script \[0K\[0K\[36;1mPreparing environment\[0;m\[0;m Running on RUNNER-YHWVQM67 via prd-win-grun-04... section_end:1729568789:prepare_script \[0Ksection_start:1729568789:get_sources \[0K\[0K\[36;1mGetting source from Git repository\[0;m\[0;m \[32;1mFetching changes with git depth set to 20...\[0;m Initialized empty Git repository in C:/builds/REDACTED/.git/ \[32;1mCreated fresh repository.\[0;m \[32;1mChecking out 61fe628c as detached HEAD (ref is feature/DEVOPS-1379-runner-issue)...\[0;m git-lfs/3.5.1 (GitHub; windows amd64; go 1.21.7; git e237bb3a)
\[32;1mUpdating/initializing submodules recursively with git depth set to 20...\[0;m \[32;1mUpdated submodules\[0;m git-lfs/3.5.1 (GitHub; windows amd64; go 1.21.7; git e237bb3a) section_end:1729568815:get_sources \[0Ksection_start:1729568815:step_script \[0K\[0K\[36;1mExecuting "step_script" stage of the job script\[0;m\[0;m \[0KUsing docker image sha256:5dab0bd985d90e5637878b8f972ee0e229f3fa9f790299e82895a5b407d34bca for mcr.microsoft.com/dotnet/framework/sdk:4.8 with digest mcr.microsoft.com/dotnet/framework/sdk@sha256:96f6d277d5d574fce9fc13d5edc110db7809ab11c1bbace1be2f3bcf5e0032e2 ...\[0;m \[32;1m$ dotnet nuget update source REDACTED.io-internal -u $PROGET_USER -p <span dir="">PROGET_PASSWORD --valid-authentication-types "basic" --configfile "nuget.config" --store-password-in-clear-text\[0;m Package source "REDACTED.io-internal" was successfully updated. \[32;1m</span> dotnet nuget update source REDACTED.io -u $PROGET_USER -p <span dir="">PROGET_PASSWORD --valid-authentication-types "basic" --configfile "nuget.config" --store-password-in-clear-text\[0;m Package source "REDACTED.io" was successfully updated. \[32;1m</span> dotnet --info\[0;m
Environment description
We are using a custom install for a standalone GitLab instance.
The runners in question are using gitlab-runner-windows-amd64.exe
on Windows Servers 2022 (21H2) using Docker version 27.2.0, build 3ab4256
We're using the docker-windows runner.
config.toml contents
concurrent = 2
check_interval = 0
shutdown_timeout = 0
[session_server]
session_timeout = 1800
[[runners]]
name = "prd-win-grun-04"
url = "https://gitlab.REDACTED"
id = 138
token = "REDACTED"
token_obtained_at = 2024-09-06T01:03:50Z
token_expires_at = 0001-01-01T00:00:00Z
executor = "docker-windows"
shell = "powershell"
[runners.custom_build_dir]
[runners.cache]
Type = "s3"
Path = "windows"
Shared = true
[runners.cache.s3]
ServerAddress = "s3.amazonaws.com"
AccessKey = "REDACTED"
SecretKey = "REDACTED"
BucketName = "REDACTED-cache"
BucketLocation = "REDACTED"
Insecure = false
ServerSideEncryption = "KMS"
ServerSideEncryptionKeyID = "alias/aws/s3"
[runners.docker]
tls_verify = false
image = "mcr.microsoft.com/dotnet/sdk:6.0-windowsservercore-ltsc2022"
cpus = "2"
privileged = false
disable_entrypoint_overwrite = false
oom_kill_disable = false
disable_cache = true
shm_size = 0
pull_policy = ["if-not-present"]
Used GitLab Runner version
Version: 17.5.0
Git revision: 66a723c3
Git branch: 17-5-stable
GO version: go1.22.7
Built: 2024-10-17T15:26:50+0000
OS/Arch: windows/amd64
Possible fixes
Workarounds as of 2024-10-25
- Upgrade to GitLab Runner 17.5.2
- Set the
FF_NETWORK_PER_BUILD
feature flag. - Set a
network_mode
in the runner config.