Hangs between stages on docker-windows on Windows 11 with KB5124008

Summary

So after KB5124008 (and now, KB5129195), our Windows docker jobs got a lot slower. It seems that for some reason there's now a 2 minute delay between every runner stage (prepare, cache, artifact, run, etc etc). This adds at about 20 minutes to every Windows job we have.

Steps to reproduce

1. Install KB5124008 update.
2. Observe hangs.
3. Uninstall KB5124008.
4. Observe lack of delays.

Actual behavior

Approximately 2 minutes of delays between stages.

Expected behavior

No delays

The following is the job start up log with KB5124008 installed:

Start of raw job log

2026-09-15T15:21:53.149565Z 00O Running with gitlab-runner 19.3.1 (a16f5092)
2026-09-15T15:21:53.149565Z 00O   on docker-windows on win-builder04 mAH9SttBx, system ID: s_5c5456080a3e
2026-09-15T15:21:53.149565Z 00O   feature flags: FF_SCRIPT_SECTIONS:true
2026-09-15T15:21:53.149565Z 00O Resolving secrets
2026-09-15T15:21:53.149565Z 00O section_start:1789485713:prepare_executor
2026-09-15T15:21:53.149565Z 00O+Preparing the "docker-windows" executor
2026-09-15T15:21:53.152757Z 00O Using Docker executor with image gitlab.example.com:5050/img-ci-windows:ltsc2025-2026-07-01 ...
2026-09-15T15:21:53.154347Z 00O Using effective pull policy of [always] for container gitlab.example.com:5050/img-ci-windows:ltsc2025-2026-07-01
2026-09-15T15:21:53.155939Z 00O Authenticating with credentials from job payload (GitLab Registry)
2026-09-15T15:21:53.155939Z 00O Pulling docker image gitlab.example.com:5050/img-ci-windows:ltsc2025-2026-07-01 ...
2026-09-15T15:21:53.392905Z 00O Using docker image sha256:7b24bc1d549d2420a451f608287ce243cf753f2d168f435eafc98d0d89322d5f for gitlab.example.com:5050/img-ci-windows:ltsc2025-2026-07-01 with digest gitlab.example.com:5050/img-ci-windows@sha256:365a1a239d2c088c66bcfa9990648d5392a3f15d05078e3e170d92690fa4679f ...
2026-09-15T15:21:53.392905Z 00O section_end:1789485713:prepare_executor
2026-09-15T15:21:53.392905Z 00O+section_start:1789485713:prepare_script
2026-09-15T15:21:53.392905Z 00O+Preparing environment
2026-09-15T15:21:53.393982Z 00O Using effective pull policy of [always] for container sha256:cfb7a9c87ead79b31543e460ab2571369d66ea049e8143668a48417e7569f6a8
2026-09-15T15:24:16.461395Z 01O Running on RUNNER-MAH9STTB via 
2026-09-15T15:24:16.461395Z 01O win-builder04...
2026-09-15T15:24:17.187494Z 00O section_end:1789485857:prepare_script
2026-09-15T15:24:17.188054Z 00O+section_start:1789485857:get_sources
2026-09-15T15:24:17.188054Z 00O+Getting source from Git repository
2026-09-15T15:26:39.797453Z 01O $ $ if (Test-Path $env:CI_PROJECT_DIR) {
2026-09-15T15:26:39.797453Z 01O   Get-ChildItem $env:CI_PROJECT_DIR -Exclude '.git' | ForEach-Object {
2026-09-15T15:26:39.797453Z 01O     cmd /c "rd /s /q `"$($_.FullName)`"" 2>$null;
2026-09-15T15:26:39.797453Z 01O     Remove-Item -Recurse -Force $_.FullName -ErrorAction SilentlyContinue
2026-09-15T15:26:39.797453Z 01O   }
2026-09-15T15:26:39.797453Z 01O }
2026-09-15T15:26:39.797453Z 01O git config --global --add safe.directory *;
2026-09-15T15:26:39.797453Z 01O git config --global core.autocrlf input;
2026-09-15T15:26:39.797453Z 01O git config --global core.longpaths true;
2026-09-15T15:26:39.797453Z 01O cmd /c "echo 10.12.24.116 gitlab.example.com >> C:\Windows\System32\drivers\etc\hosts ";
2026-09-15T15:27:29.840998Z 01O Gitaly correlation ID: 01M2JTJ0RMM1WXWHA03NQSRMW4
2026-09-15T15:27:30.122472Z 01O Fetching changes...
2026-09-15T15:27:30.197115Z 01O Reinitialized existing Git repository in C:/builds/mAH9SttBx/0/repo/.git/
2026-09-15T15:27:30.296264Z 01O Created fresh repository.
2026-09-15T15:27:32.854070Z 01O Checking out fcabad81 as detached HEAD (ref is refs/merge-requests/4036/merge)...
2026-09-15T15:28:50.756938Z 01O git-lfs/3.7.1 (GitHub; windows amd64; go 1.25.1; git b84b3384)
2026-09-15T15:28:52.445337Z 01O 
2026-09-15T15:28:52.445337Z 01O Skipping Git submodules setup
2026-09-15T15:28:54.044370Z 00O section_end:1789486134:get_sources
2026-09-15T15:28:54.045412Z 00O+section_start:1789486134:restore_cache
2026-09-15T15:28:54.045412Z 00O+Restoring cache
2026-09-15T15:31:16.361705Z 01O Version:      19.3.1
2026-09-15T15:31:16.361705Z 01O Git revision: a16f5092
2026-09-15T15:31:16.361705Z 01O Git branch:   19-3-stable
2026-09-15T15:31:16.361705Z 01O GO version:   go1.26.5
2026-09-15T15:31:16.361705Z 01O Built:        2026-08-24T14:27:08Z
2026-09-15T15:31:16.361705Z 01O OS/Arch:      windows/amd64
2026-09-15T15:31:16.371748Z 01O Checking cache for build-cache-build-windows-7e86ff06e32dbb82d39644422480529cae98d9ae-2-protected...
2026-09-15T15:31:16.465220Z 01E No URL provided, cache will not be downloaded from shared cache server. Instead a local version of cache will be extracted. 
2026-09-15T15:32:04.195043Z 01O Successfully extracted cache
2026-09-15T15:32:04.296898Z 01O Version:      19.3.1
2026-09-15T15:32:04.296898Z 01O Git revision: a16f5092
2026-09-15T15:32:04.296898Z 01O Git branch:   19-3-stable
2026-09-15T15:32:04.296898Z 01O GO version:   go1.26.5
2026-09-15T15:32:04.296898Z 01O Built:        2026-08-24T14:27:08Z
2026-09-15T15:32:04.296898Z 01O OS/Arch:      windows/amd64
2026-09-15T15:32:04.299171Z 01O Checking cache for default-cache-build-windows-20250623-2-protected...
2026-09-15T15:32:04.404885Z 01E No URL provided, cache will not be downloaded from shared cache server. Instead a local version of cache will be extracted. 
2026-09-15T15:32:04.414797Z 01O Successfully extracted cache
2026-09-15T15:32:04.928002Z 00O section_end:1789486324:restore_cache
2026-09-15T15:32:04.928558Z 00O+section_start:1789486324:step_script
2026-09-15T15:32:04.928558Z 00O Using effective pull policy of [always] for container gitlab.example.com:5050/img-ci-windows:ltsc2025-2026-07-01
2026-09-15T15:32:04.930138Z 00O Using docker image sha256:7b24bc1d549d2420a451f608287ce243cf753f2d168f435eafc98d0d89322d5f for gitlab.example.com:5050/img-ci-windows:ltsc2025-2026-07-01 with digest gitlab.example.com:5050/img-ci-windows@sha256:365a1a239d2c088c66bcfa9990648d5392a3f15d05078e3e170d92690fa4679f ...
2026-09-15T15:32:08.636336Z 01O $ $ cmd /c "echo 10.12.24.116 gitlab.example.com >> C:\Windows\System32\drivers\etc\hosts ";
2026-09-15T15:32:08.636336Z 01O cmd /c "echo 10.12.24.121 nexus.example.com >> C:\Windows\System32\drivers\etc\hosts ";

Note the large jumps in timestamps (other than the git repo clean/checkout) - it took 9 minutes to get to the actual start of the job

The end of the job log has similar delays with the cache/artifact taking several minutes to even start.

(the pre-get_sources script is admittedly significant, but it's the time the script actually takes to start running here that's important)

Start of raw job log

2026-09-15T17:11:12.782455Z 00O Running with gitlab-runner 19.3.1 (a16f5092)
2026-09-15T17:11:12.782455Z 00O   on docker-windows on win-builder04 mAH9SttBx, system ID: s_5c5456080a3e
2026-09-15T17:11:12.782455Z 00O   feature flags: FF_SCRIPT_SECTIONS:true
2026-09-15T17:11:12.782455Z 00O Resolving secrets
2026-09-15T17:11:12.783023Z 00O section_start:1789492272:prepare_executor
2026-09-15T17:11:12.783023Z 00O+Preparing the "docker-windows" executor
2026-09-15T17:11:12.785699Z 00O Using Docker executor with image gitlab.example.com:5050/img-ci-windows:ltsc2022-2026-07-01 ...
2026-09-15T17:11:12.787843Z 00O Using effective pull policy of [always] for container gitlab.example.com:5050/img-ci-windows:ltsc2022-2026-07-01
2026-09-15T17:11:12.789435Z 00O Authenticating with credentials from job payload (GitLab Registry)
2026-09-15T17:11:12.789435Z 00O Pulling docker image gitlab.example.com:5050/img-ci-windows:ltsc2022-2026-07-01 ...
2026-09-15T17:11:13.032797Z 00O Using docker image sha256:705b6cbb857afe34d4d3448e614e03d1d5c4087c26ca4e0b613ec383f46d96c4 for gitlab.example.com:5050/img-ci-windows:ltsc2022-2026-07-01 with digest gitlab.example.com:5050/img-ci-windows@sha256:80d55a7666559ead2e35eaa36fd9e7fd4808c660b973bcfe973dcf2b9cf08199 ...
2026-09-15T17:11:13.032797Z 00O section_end:1789492273:prepare_executor
2026-09-15T17:11:13.035426Z 00O+section_start:1789492273:prepare_script
2026-09-15T17:11:13.035426Z 00O+Preparing environment
2026-09-15T17:11:13.037030Z 00O Using effective pull policy of [always] for container sha256:cfb7a9c87ead79b31543e460ab2571369d66ea049e8143668a48417e7569f6a8
2026-09-15T17:11:17.272668Z 01O Running on RUNNER-MAH9STTB via 
2026-09-15T17:11:17.272668Z 01O win-builder04...
2026-09-15T17:11:18.755881Z 00O section_end:1789492278:prepare_script

2026-09-15T17:11:18.758054Z 00O+section_start:1789492278:get_sources

2026-09-15T17:11:18.758054Z 00O+Getting source from Git repository
2026-09-15T17:11:21.574428Z 01O $ $ if (Test-Path $env:CI_PROJECT_DIR) {
2026-09-15T17:11:21.574428Z 01O   Get-ChildItem $env:CI_PROJECT_DIR -Exclude '.git' | ForEach-Object {
2026-09-15T17:11:21.574428Z 01O     cmd /c "rd /s /q `"$($_.FullName)`"" 2>$null;
2026-09-15T17:11:21.574428Z 01O     Remove-Item -Recurse -Force $_.FullName -ErrorAction SilentlyContinue
2026-09-15T17:11:21.574428Z 01O   }
2026-09-15T17:11:21.574428Z 01O }
2026-09-15T17:11:21.574428Z 01O git config --global --add safe.directory *;
2026-09-15T17:11:21.574428Z 01O git config --global core.autocrlf input;
2026-09-15T17:11:21.574428Z 01O git config --global core.longpaths true;
2026-09-15T17:11:21.574428Z 01O cmd /c "echo 10.12.24.116 gitlab.example.com >> C:\Windows\System32\drivers\etc\hosts ";
2026-09-15T17:12:30.825224Z 01O Gitaly correlation ID: 01M2K0T6Z6TB6DXS7GXF0YEF14
2026-09-15T17:12:31.133479Z 01O Fetching changes...
2026-09-15T17:12:31.211419Z 01O Reinitialized existing Git repository in C:/builds/mAH9SttBx/0/repo/.git/
2026-09-15T17:12:31.322295Z 01O Created fresh repository.
2026-09-15T17:12:33.514322Z 01O Checking out 3a9729b4 as detached HEAD (ref is refs/merge-requests/4708/merge)...
2026-09-15T17:14:10.297183Z 01O git-lfs/3.7.1 (GitHub; windows amd64; go 1.25.1; git b84b3384)
2026-09-15T17:14:12.124800Z 01O 
2026-09-15T17:14:12.124800Z 01O Skipping Git submodules setup
2026-09-15T17:14:14.585084Z 00O section_end:1789492454:get_sources
2026-09-15T17:14:14.587199Z 00O+section_start:1789492454:restore_cache
2026-09-15T17:14:14.587199Z 00O+Restoring cache
2026-09-15T17:14:17.080403Z 01O Version:      19.3.1
2026-09-15T17:14:17.080403Z 01O Git revision: a16f5092
2026-09-15T17:14:17.080403Z 01O Git branch:   19-3-stable
2026-09-15T17:14:17.080403Z 01O GO version:   go1.26.5
2026-09-15T17:14:17.080403Z 01O Built:        2026-08-24T14:27:08Z
2026-09-15T17:14:17.080403Z 01O OS/Arch:      windows/amd64
2026-09-15T17:14:17.091436Z 01O Checking cache for build-cache-test-rtt-framework-windows-48543ccf9d2da92fb8ed381b2bcf0ce8259b6b81-2-protected...
2026-09-15T17:14:17.201480Z 01E No URL provided, cache will not be downloaded from shared cache server. Instead a local version of cache will be extracted. 
2026-09-15T17:14:17.202533Z 01E WARNING: Cache file does not exist                 
2026-09-15T17:14:17.222949Z 01O Failed to extract cache
2026-09-15T17:14:17.321429Z 01O Version:      19.3.1
2026-09-15T17:14:17.321429Z 01O Git revision: a16f5092
2026-09-15T17:14:17.321429Z 01O Git branch:   19-3-stable
2026-09-15T17:14:17.321429Z 01O GO version:   go1.26.5
2026-09-15T17:14:17.321429Z 01O Built:        2026-08-24T14:27:08Z
2026-09-15T17:14:17.321429Z 01O OS/Arch:      windows/amd64
2026-09-15T17:14:17.329394Z 01O Checking cache for default-cache-test-rtt-framework-windows-20250623-2-protected...
2026-09-15T17:14:17.431935Z 01E No URL provided, cache will not be downloaded from shared cache server. Instead a local version of cache will be extracted. 
2026-09-15T17:14:17.443267Z 01O Successfully extracted cache
2026-09-15T17:14:18.701349Z 00O section_end:1789492458:restore_cache

2026-09-15T17:14:18.701886Z 00O+section_start:1789492458:download_artifacts
2026-09-15T17:14:18.701886Z 00O+Downloading artifacts
2026-09-15T17:14:20.996863Z 01O Version:      19.3.1
2026-09-15T17:14:20.996863Z 01O Git revision: a16f5092
2026-09-15T17:14:20.996863Z 01O Git branch:   19-3-stable
2026-09-15T17:14:20.996863Z 01O GO version:   go1.26.5
2026-09-15T17:14:20.996863Z 01O Built:        2026-08-24T14:27:08Z
2026-09-15T17:14:20.996863Z 01O OS/Arch:      windows/amd64
2026-09-15T17:14:21.009269Z 01O Downloading artifacts for Installer:Windows (910212)...
2026-09-15T17:14:34.930941Z 01E Downloading artifacts from coordinator... ok        correlation_id=01M2K0ZZM201508NMPF8QDTE57 id=910212 status=200 token=68_rz-T57
2026-09-15T17:14:42.747833Z 00O section_end:1789492482:download_artifacts
2026-09-15T17:14:42.748370Z 00O+section_start:1789492482:step_script
2026-09-15T17:14:42.748370Z 00O+Executing "step_script" stage of the job script
2026-09-15T17:14:42.748370Z 00O Using effective pull policy of [always] for container gitlab.example.com:5050/img-ci-windows:ltsc2022-2026-07-01
2026-09-15T17:14:42.749426Z 00O Using docker image sha256:705b6cbb857afe34d4d3448e614e03d1d5c4087c26ca4e0b613ec383f46d96c4 for gitlab.example.com:5050/img-ci-windows:ltsc2022-2026-07-01 with digest gitlab.example.com:5050/img-ci-windows@sha256:80d55a7666559ead2e35eaa36fd9e7fd4808c660b973bcfe973dcf2b9cf08199 ...
2026-09-15T17:14:50.511975Z 01O $ $ cmd /c "echo 10.12.24.116 gitlab.example.com >> C:\Windows\System32\drivers\etc\hosts ";
2026-09-15T17:14:50.511975Z 01O cmd /c "echo 10.12.24.121 nexus.example.com >> C:\Windows\System32\drivers\etc\hosts ";

Note the very small amount of time taken between stages

Relevant logs and/or screenshots

I've also done some logging with procmon, just to see if I could see anything. I couldn't - as far as I can tell, gitlab-runner/dockerd/vmcompute just sit there doing very little (gitlab-runner itself does seem to occasionally ping the gitlab server), until cExecSrv.exe starts. While I still have the PML recording I can't share that and I can't fit anything of relevance in a single screenshot.

Environment description

Client:
 Version:    29.8.0
 Context:    default
 Debug Mode: false

Server:
 Containers: 1
  Running: 0
  Paused: 0
  Stopped: 1
 Images: 2
 Server Version: 29.8.0
 Storage Driver: windowsfilter
  Windows:
 Logging Driver: json-file
 Plugins:
  Volume: local
  Network: ics internal l2bridge l2tunnel nat null overlay private transparent
  Log: awslogs etwlogs fluentd gcplogs gelf json-file local splunk syslog
 CDI spec directories:
  /etc/cdi
  /var/run/cdi
 Swarm: inactive
 Default Isolation: process
 Kernel Version: 10.0 26100 (26100.1.amd64fre.ge_release.240331-1435)
 Operating System: Microsoft Windows Version 24H2 (OS Build 26100.9168)
 OSType: windows
 Architecture: x86_64
 CPUs: 32
 Total Memory: 31.9GiB
 Name: win-builder04
 ID: 7bf0d023-738f-4799-9779-82b4b948bf39
 Docker Root Dir: C:\ProgramData\docker
 Debug Mode: false
 Experimental: false
 Insecure Registries:
  ::1/128
  127.0.0.0/8
 Live Restore Enabled: false
 Product License: Community Engine

Process isolation is used.

I'm not actually certain if gitlab-runner is at fault here, but I cannot reproduce any sort of delay when running docker run manually (interactive or not), so I have to start somewhere.

Given the issue occurred with KB5124008 and all the other issues with that update I'd be prepared to bet that this is actually a Windows bug, but I've not seen anyone else mention anything vaguely related (there is something to do with WSL & shared folders, but I don't believe that's relevant here and besides it was supposed to be fixed by KB5129195 and it hasn't helped here)

config.toml contents

concurrent = 1
check_interval = 0
listen_address = "0.0.0.0:9252"
log_format = "text"
[session_server]
  listen_address = "0.0.0.0:8093"
  advertise_address = "win-builder04.example.com:8093"
  session_timeout = 1800

[[runners]]
  name = "docker-windows on win-builder04"
  limit = 0
  url = "https://gitlab.example.com"
  environment = []
  pre_get_sources_script = "if (Test-Path $env:CI_PROJECT_DIR) {\n  Get-ChildItem $env:CI_PROJECT_DIR -Exclude \u0027.git\u0027 | ForEach-Object {\n    cmd /c \"rd /s /q `\"$($_.FullName)`\"\" 2\u003e$null;\n    Remove-Item -Recurse -Force $_.FullName -ErrorAction SilentlyContinue\n  }\n}\ngit config --global --add safe.directory *;\ngit config --global core.autocrlf input;\ngit config --global core.longpaths true;\ncmd /c \"echo 10.12.24.116 gitlab.example.com \u003e\u003e C:\\Windows\\System32\\drivers\\etc\\hosts \";\n"
  pre_build_script  = "cmd /c \"echo 10.12.24.116 gitlab.example.com \u003e\u003e C:\\Windows\\System32\\drivers\\etc\\hosts \";\ncmd /c \"echo 10.12.24.121 nexus.example.com \u003e\u003e C:\\Windows\\System32\\drivers\\etc\\hosts \";\n"
  executor = "docker-windows"
  shell   = "pwsh"
  output_limit = 20480
  id = 0
  token = "snip"
  token_expires_at = 0001-01-01T00:00:00Z
  [runners.docker]
    image = "mcr.microsoft.com/windows/servercore:ltsc2019"
    helper_image = ""
    shm_size = 0
    pull_policy = ["always"]
    volumes = ["C:\\builds:C:\\builds", "C:\\cache:C:\\cache", "C:\\ProgramData\\foo:C:\\ProgramData\\foo:ro"]
    extra_hosts = ["gitlab.example.com:10.12.24.116", "nexus.example.com:10.12.24.121"]
    isolation = "process"
    [runners.docker.sysctls]
      "fs.mqueue.msg_max" = "2048"
  [runners.cache]

Used GitLab Runner version

Version:      19.3.1
Git revision: a16f5092
Git branch:   19-3-stable
GO version:   go1.26.5
Built:        2026-08-24T14:27:08Z
OS/Arch:      windows/amd64

-->

Possible fixes

The only work around I've found is to uninstall KB5124008 & KB5129195. Given the number of CVEs fixed, this is not especially feasible medium-to-long term.