Characters randomly vanish in CI script on Windows+docker+powershell
Summary
Single characters in the CI script generated by gitlab-runner and passed to a docker executor with powershell randomly go missing.*
Steps to reproduce
- Use gitlab-runner on Windows, with a docker executor, running powershell in an image derived from
mcr.microsoft.com/windows/servercore:ltsc2019
(other shells/executors/images may be affected too, I didn't test them) - Run any CI job
Example .gitlab-ci.yml (all job scripts are affected, not only the given example)
myjob:
stage: package
variables:
PERSISTENT_PATH: C:\buildpersistent\${CI_COMMIT_BRANCH}
...
script:
- ...
- if ( Test-Path "$env:PERSISTENT_PATH" ) { Copy-Item -Path "build\output\setup" -Destination "$env:PERSISTENT_PATH" -Recurse -Force }
tags:
- windows-docker
Actual behavior
Example job output:
$ if ( Test-Path "$env:PERSISTENT_PATH" ) { Copy-Item -Path "build\output\setup" -Destination "$env:PERSISTENT_PATH" -Recurse -Force }
Test-Path : Cannot bind argument to parameter 'Path' because it is an empty
string.
At line:1 char:16
+ if ( Test-Path "$env:PERSISENT_PATH" ) { Copy-Item -Path "build\outpu ...
+ ~~~~~~~~~~~~~~~~~~~~~
+ CategoryInfo : InvalidData: (:) [Test-Path], ParentContainsErro
rRecordException
+ FullyQualifiedErrorId : ParameterArgumentValidationErrorEmptyStringNotAl
lowed,Microsoft.PowerShell.Commands.TestPathCommand
Notice the missing T
- the error message shows PERSISENT_PATH
, whereas the previously logged executed command mentions the correct PERSISTENT_PATH
. Note that these are two different locations in the generated script - gitlab-runner first echo
es the command about to be executed, then executes it. Only the executed version is affected here. Since the location at which a character disappears is random, it may very well also be the other way around.
Here are my findings so far:
- Any single character can randomly disappear
- When retrying the job, the same character disappears (deterministic behavior)
- When adding any commit, or when running the same pipeline as a different gitlab user, the bug will be gone or appear at another location in the script (random behavior)
- Both the
script
in the.gitlab-ci.yml
as well as commands inserted by gitlab-runner can be affected
Expected behavior
Please keep all characters in the script.
Environment description
- Using Gitlab-Runner 13.7.0 on Windows Server Core LTSC 2019, registered as project-specific runner
config.toml contents
concurrent = 1
check_interval = 10
[session_server]
session_timeout = 1800
[[runners]]
name = "myrunner"
url = "https://example.com/"
token = "<token>"
output_limit = 32768
executor = "docker-windows"
[runners.docker]
tls_verify = false
image = "mcr.microsoft.com/windows/nanoserver:1809"
privileged = false
disable_entrypoint_overwrite = false
oom_kill_disable = false
disable_cache = false
volumes = ["c:\\cache", "C:\\buildpersistent:C:\\buildpersistent"]
shm_size = 0
pull_policy = "if-not-present"
Additional runners have been removed from the config for breviety
- Using Docker version 19.03.14, build e820475
docker info
Client:
Debug Mode: false
Plugins:
cluster: Manage Mirantis Container Cloud clusters (Mirantis Inc., v1.9.0)
Server:
Containers: 0
Running: 0
Paused: 0
Stopped: 0
Images: 32
Server Version: 19.03.14
Storage Driver: windowsfilter
Windows:
Logging Driver: json-file
Plugins:
Volume: local
Network: ics internal l2bridge l2tunnel nat null overlay private transparent
Log: awslogs etwlogs fluentd gcplogs gelf json-file local logentries splunk syslog
Swarm: inactive
Default Isolation: process
Kernel Version: 10.0 17763 (17763.1.amd64fre.rs5_release.180914-1434)
Operating System: Windows Server 2019 Datacenter Version 1809 (OS Build 17763.1339)
OSType: windows
Architecture: x86_64
CPUs: 14
Total Memory: 12GiB
Name: redacted
ID: redacted
Docker Root Dir: C:\ProgramData\docker
Debug Mode: false
Registry: https://index.docker.io/v1/
Labels:
Experimental: false
Insecure Registries:
127.0.0.0/8
Live Restore Enabled: false
Example Dockerfile
# escape=`
FROM mcr.microsoft.com/windows/servercore:ltsc2019
SHELL ["cmd", "/S", "/C"]
# ...
ENTRYPOINT ["C:\\BuildTools\\VC\\Auxiliary\\Build\\vcvars64.bat", "-vcvars_spectre_libs=spectre", "&&"]
CMD ["powershell"]
Used GitLab Runner version
Version: 13.7.0
Git revision: 943fc252
Git branch: 13-7-stable
GO version: go1.13.8
Built: 2020-12-21T13:47:18+0000
OS/Arch: windows/amd64
* This bug has made me question my sanity for a considerable amount of time.