Skip to content

Artifacts uploaded by Windows runners are silently missing files/directories in long paths (Can be fixed by compiling gitlab-runner.exe with Go1.17 or later)

Summary

After a build, gitlab-runner compresses artifacts and uploads them to the Gitlab server.
On Windows when files/directories to be included in the artifact have a long path (around 260 characters) those files/directories are missing from the artifact.
No warnings or errors are produced during the build(upload).

When gitlab-runner.exe is compiled with Go1.17 or later, files/directories with long paths are correctly uploaded.
The Windows gitlab-runner.exe release 14.6.0 is compiled with Go1.13.8 so it is affected by the problem.
The Windows gitlab-runner.exe release 14.7.0 is compiled with Go1.17.5 so it is not affected by the problem.

Go Type Result
go1.17.5 self compiled based on 14.6.0 OK
go1.17 self compiled based on 14.6.0 OK
go1.16.12 self compiled based on 14.6.0 NOT OK
go1.15.15 self compiled based on 14.6.0 NOT OK
go1.13.15 self compiled based on 14.6.0 NOT OK
go1.13.8 downloaded official 14.6.0 NOT OK
go1.17.5 downloaded official 14.7.0 OK

Steps to reproduce

Gitlab runner on running on Windows with Powershell.

.gitlab-ci.yml 
build:
  stage: build
  script:  
  # run PowerShell script
  - powershell -File build.ps1
  artifacts:
    paths:
      - result
    expire_in: 1 day
.build.ps1 
$l = Get-Location
# Convert to String
$p=""+$l+"\result"

New-Item -ItemType Directory -Force -Path $p
Set-Content "${p}\test.txt" 'Example contents'

$dirs = New-Object Collections.Generic.List[string]
# First create short directories and remember them
foreach ($i in 1..15){
	$p = $p + "\" + $i
	Write-Host Creating dir $i 
	New-Item -ItemType Directory -Force -Path $p
	Set-Content "${p}\test.txt" 'Example contents'
	$dirs.Insert(0, $p)
}

# Rename short directories to longer name
foreach ($short in $dirs){
	$long=$short+"_looooo0000oooong"
	Write-Host Renaming dir $short -> $long
	Rename-Item $short $long
}

Actual behavior

With every current release of gitlab-runner, artifacts in deeply nested directories/long paths are not uploaded correctly (files/directories with long paths are simply missing from the artifact).
No warning/error is produced about the files missing from the artifact.

Expected behavior

Artifacts are uploaded without any files missing in case of deeply nested directories / long paths.

Relevant logs and/or screenshots

Bad job (only 18 files/directories found) 
Running with gitlab-runner 14.6.0 (5316d4ac)
...
Uploading artifacts for successful job 00:03
Version:      14.6.0
Git revision: 5316d4ac
Git branch:   14-6-stable
GO version:   go1.13.8
Built:        2021-12-17T17:35:49+0000
OS/Arch:      windows/386
Uploading artifacts...
Runtime platform                                    arch=386 os=windows pid=17704 revision=5316d4ac version=14.6.0
result: found 18 matching files and directories    
Uploading artifacts as "archive" to coordinator... ok  id=1980101530 responseStatus=201 Created token=Wg1eyzsN
Cleaning up project directory and file based variables 00:00
Job succeeded
Good job (all 32 files/directories found) 
Running with gitlab-runner development version (HEAD)
...
Uploading artifacts for successful job 00:01
Version:      development version
Git revision: HEAD
Git branch:   HEAD
GO version:   go1.17
Built:        unknown
OS/Arch:      windows/amd64
Uploading artifacts...
Runtime platform                                    arch=amd64 os=windows pid=11020 revision=HEAD version=development version
result: found 32 matching files and directories    
Uploading artifacts as "archive" to coordinator... ok  id=1980150789 responseStatus=201 Created token=2rSVDC8x
Cleaning up project directory and file based variables 00:01
Job succeeded

Environment description

The problem of missing files happens when running gitlab-runner.exe against Gitlab.com and when running it against my own server.
Windows 10 latest version 21H2 19044.1415 (But earlier Windows 10 versions behave the same way)

config.toml contents 
concurrent = 1
check_interval = 0

[session_server]
  session_timeout = 1800

[[runners]]
  name = "test_upload_long_path"
  url = "https://gitlab.com/"
  token = "xxxxxx"
  executor = "shell"
  shell = "powershell"
  [runners.custom_build_dir]
  [runners.cache]
    [runners.cache.s3]
    [runners.cache.gcs]
    [runners.cache.azure]

Used GitLab Runner version

All gitlab-runner.exe versions compiled with Go1.16 or earlier are affected.
All gitlab-runner.exe versions compiled with Go1.17 or later are NOT affected.
The problem affects the currentlatest gitlab-runner.exe version 14.6.0 as it is compiled with Go1.13.8.

Possible fixes

Using Go1.17 (or later) to compile gitlab-runner.exe seems to fix the problem.
There is no entry related to this change in the Go release notes but it was likely caused by commit runtime: support long paths without fixup on Windows 10 >= 1607

This is related to !2294 (closed)
This is related to #3690 (closed)

Edited by Peter Oberndorfer