Skip to content

Characters randomly vanish in CI script on Windows+docker+powershell

Summary

Single characters in the CI script generated by gitlab-runner and passed to a docker executor with powershell randomly go missing.*

Steps to reproduce

  • Use gitlab-runner on Windows, with a docker executor, running powershell in an image derived from mcr.microsoft.com/windows/servercore:ltsc2019 (other shells/executors/images may be affected too, I didn't test them)
  • Run any CI job
Example .gitlab-ci.yml (all job scripts are affected, not only the given example)
myjob:
  stage: package
  variables:
    PERSISTENT_PATH: C:\buildpersistent\${CI_COMMIT_BRANCH}
    ...
  script:
    - ...
    - if ( Test-Path "$env:PERSISTENT_PATH" ) { Copy-Item -Path "build\output\setup" -Destination "$env:PERSISTENT_PATH" -Recurse -Force }
  tags:
    - windows-docker

Actual behavior

Example job output:

$ if ( Test-Path "$env:PERSISTENT_PATH" ) { Copy-Item -Path "build\output\setup" -Destination "$env:PERSISTENT_PATH" -Recurse -Force }
Test-Path : Cannot bind argument to parameter 'Path' because it is an empty 
string.
At line:1 char:16
+ if ( Test-Path "$env:PERSISENT_PATH" ) { Copy-Item -Path "build\outpu ...
+                ~~~~~~~~~~~~~~~~~~~~~
    + CategoryInfo          : InvalidData: (:) [Test-Path], ParentContainsErro 
   rRecordException
    + FullyQualifiedErrorId : ParameterArgumentValidationErrorEmptyStringNotAl 
   lowed,Microsoft.PowerShell.Commands.TestPathCommand

Notice the missing T - the error message shows PERSISENT_PATH, whereas the previously logged executed command mentions the correct PERSISTENT_PATH. Note that these are two different locations in the generated script - gitlab-runner first echoes the command about to be executed, then executes it. Only the executed version is affected here. Since the location at which a character disappears is random, it may very well also be the other way around.

Here are my findings so far:

  • Any single character can randomly disappear
  • When retrying the job, the same character disappears (deterministic behavior)
  • When adding any commit, or when running the same pipeline as a different gitlab user, the bug will be gone or appear at another location in the script (random behavior)
  • Both the script in the .gitlab-ci.yml as well as commands inserted by gitlab-runner can be affected

Expected behavior

Please keep all characters in the script.

Environment description

  • Using Gitlab-Runner 13.7.0 on Windows Server Core LTSC 2019, registered as project-specific runner
config.toml contents
concurrent = 1
check_interval = 10

[session_server]
  session_timeout = 1800

[[runners]]
  name = "myrunner"
  url = "https://example.com/"
  token = "<token>"
  output_limit = 32768
  executor = "docker-windows"
  [runners.docker]
    tls_verify = false
    image = "mcr.microsoft.com/windows/nanoserver:1809"
    privileged = false
    disable_entrypoint_overwrite = false
    oom_kill_disable = false
    disable_cache = false
    volumes = ["c:\\cache", "C:\\buildpersistent:C:\\buildpersistent"]
    shm_size = 0
    pull_policy = "if-not-present"

Additional runners have been removed from the config for breviety

  • Using Docker version 19.03.14, build e820475
docker info
Client: 
 Debug Mode: false 
 Plugins:
  cluster: Manage Mirantis Container Cloud clusters (Mirantis Inc., v1.9.0)

Server:
 Containers: 0
  Running: 0
  Paused: 0
  Stopped: 0
 Images: 32
 Server Version: 19.03.14
 Storage Driver: windowsfilter
  Windows:
 Logging Driver: json-file
 Plugins:
  Volume: local
  Network: ics internal l2bridge l2tunnel nat null overlay private transparent
  Log: awslogs etwlogs fluentd gcplogs gelf json-file local logentries splunk syslog
 Swarm: inactive
 Default Isolation: process
 Kernel Version: 10.0 17763 (17763.1.amd64fre.rs5_release.180914-1434)
 Operating System: Windows Server 2019 Datacenter Version 1809 (OS Build 17763.1339)
 OSType: windows
 Architecture: x86_64
 CPUs: 14
 Total Memory: 12GiB
 Name: redacted
 ID: redacted
 Docker Root Dir: C:\ProgramData\docker
 Debug Mode: false
 Registry: https://index.docker.io/v1/
 Labels:
 Experimental: false
 Insecure Registries:
  127.0.0.0/8
 Live Restore Enabled: false
Example Dockerfile
# escape=`
FROM mcr.microsoft.com/windows/servercore:ltsc2019
SHELL ["cmd", "/S", "/C"]
# ...
ENTRYPOINT ["C:\\BuildTools\\VC\\Auxiliary\\Build\\vcvars64.bat", "-vcvars_spectre_libs=spectre", "&&"]
CMD ["powershell"]

Used GitLab Runner version

Version:      13.7.0 
Git revision: 943fc252
Git branch:   13-7-stable
GO version:   go1.13.8
Built:        2020-12-21T13:47:18+0000
OS/Arch:      windows/amd64

* This bug has made me question my sanity for a considerable amount of time.