Cache extraction fails with FATAL: remove directory not empty with FF_USE_FASTZIP enabled
Summary
When creating a cache with globbing and that contains a recursive symlink, the resulting zip file will contain sub directories to the symlinked file.
For example, see this cache.zip, where test/dir/a is a symlink to ../a:
+ unzip -vql cache.zip
Length Method Size Cmpr Date Time CRC-32 Name
-------- ------ ------- ---- ---------- ----- -------- ----
0 Stored 0 0% 2021-07-29 15:51 00000000 test/dir/
0 Stored 0 0% 2021-07-29 15:51 00000000 test/dir/a/
4 Stored 4 0% 2021-07-29 15:51 7bd30558 test/dir/a/a
4 Stored 4 0% 2021-07-29 15:51 7bd30558 test/dir/a/a/a
4 Stored 4 0% 2021-07-29 15:51 7bd30558 test/dir/a/a/a/a
4 Stored 4 0% 2021-07-29 15:51 7bd30558 test/dir/a/a/a/a/aThis is not directly related to FF_USE_FASTZIP, but the extraction succeeds using the regular zip implementation, but fails with the feature flag enabled. I suppose both the extraction and cache creation issues could be considered bugs, but fixing one of them should solve the issue.
Steps to reproduce
Place this script in an empty folder and run it:
#!/bin/sh
set -ex
gitlab-runner -v
# Clean up previous runs
rm -rf source extract cache.zip
# Create folder structure for reproducing
mkdir -p source/test/dir/a extract
ln -s ../a source/test/dir/a/a
# Create cache
(cd source && gitlab-runner cache-archiver --file ../cache.zip --path 'test/**')
# Show cache contents
unzip -vql cache.zip
# Extract (which will fail)
(cd extract && FF_USE_FASTZIP=true gitlab-runner cache-extractor --file ../cache.zip)Output on my system:
❯ ./repro.sh
+ gitlab-runner -v
Version: 14.1.0
Git revision: 8925d9a0
Git branch: 14-1-stable
GO version: go1.13.8
Built: 2021-07-20T11:43:26+0000
OS/Arch: linux/amd64
+ rm -rf source extract cache.zip
+ mkdir -p source/test/dir/a extract
+ ln -s ../a source/test/dir/a/a
+ cd source
+ gitlab-runner cache-archiver --file ../cache.zip --path 'test/**'
Runtime platform arch=amd64 os=linux pid=3193860 revision=8925d9a0 version=14.1.0
test/**: found 45 matching files and directories
No URL provided, cache will be not uploaded to shared cache server. Cache will be stored only locally.
+ unzip -vql cache.zip
Length Method Size Cmpr Date Time CRC-32 Name
-------- ------ ------- ---- ---------- ----- -------- ----
0 Stored 0 0% 2021-07-29 16:22 00000000 test/dir/
0 Stored 0 0% 2021-07-29 16:22 00000000 test/dir/a/
4 Stored 4 0% 2021-07-29 16:22 7bd30558 test/dir/a/a
4 Stored 4 0% 2021-07-29 16:22 7bd30558 test/dir/a/a/a
4 Stored 4 0% 2021-07-29 16:22 7bd30558 test/dir/a/a/a/a
4 Stored 4 0% 2021-07-29 16:22 7bd30558 test/dir/a/a/a/a/a
4 Stored 4 0% 2021-07-29 16:22 7bd30558 test/dir/a/a/a/a/a/a
4 Stored 4 0% 2021-07-29 16:22 7bd30558 test/dir/a/a/a/a/a/a/a
4 Stored 4 0% 2021-07-29 16:22 7bd30558 test/dir/a/a/a/a/a/a/a/a
4 Stored 4 0% 2021-07-29 16:22 7bd30558 test/dir/a/a/a/a/a/a/a/a/a
4 Stored 4 0% 2021-07-29 16:22 7bd30558 test/dir/a/a/a/a/a/a/a/a/a/a
4 Stored 4 0% 2021-07-29 16:22 7bd30558 test/dir/a/a/a/a/a/a/a/a/a/a/a
4 Stored 4 0% 2021-07-29 16:22 7bd30558 test/dir/a/a/a/a/a/a/a/a/a/a/a/a
4 Stored 4 0% 2021-07-29 16:22 7bd30558 test/dir/a/a/a/a/a/a/a/a/a/a/a/a/a
4 Stored 4 0% 2021-07-29 16:22 7bd30558 test/dir/a/a/a/a/a/a/a/a/a/a/a/a/a/a
4 Stored 4 0% 2021-07-29 16:22 7bd30558 test/dir/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a
4 Stored 4 0% 2021-07-29 16:22 7bd30558 test/dir/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a
4 Stored 4 0% 2021-07-29 16:22 7bd30558 test/dir/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a
4 Stored 4 0% 2021-07-29 16:22 7bd30558 test/dir/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a
4 Stored 4 0% 2021-07-29 16:22 7bd30558 test/dir/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a
4 Stored 4 0% 2021-07-29 16:22 7bd30558 test/dir/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a
4 Stored 4 0% 2021-07-29 16:22 7bd30558 test/dir/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a
4 Stored 4 0% 2021-07-29 16:22 7bd30558 test/dir/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a
4 Stored 4 0% 2021-07-29 16:22 7bd30558 test/dir/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a
4 Stored 4 0% 2021-07-29 16:22 7bd30558 test/dir/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a
4 Stored 4 0% 2021-07-29 16:22 7bd30558 test/dir/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a
4 Stored 4 0% 2021-07-29 16:22 7bd30558 test/dir/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a
4 Stored 4 0% 2021-07-29 16:22 7bd30558 test/dir/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a
4 Stored 4 0% 2021-07-29 16:22 7bd30558 test/dir/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a
4 Stored 4 0% 2021-07-29 16:22 7bd30558 test/dir/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a
4 Stored 4 0% 2021-07-29 16:22 7bd30558 test/dir/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a
4 Stored 4 0% 2021-07-29 16:22 7bd30558 test/dir/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a
4 Stored 4 0% 2021-07-29 16:22 7bd30558 test/dir/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a
4 Stored 4 0% 2021-07-29 16:22 7bd30558 test/dir/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a
4 Stored 4 0% 2021-07-29 16:22 7bd30558 test/dir/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a
4 Stored 4 0% 2021-07-29 16:22 7bd30558 test/dir/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a
4 Stored 4 0% 2021-07-29 16:22 7bd30558 test/dir/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a
4 Stored 4 0% 2021-07-29 16:22 7bd30558 test/dir/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a
4 Stored 4 0% 2021-07-29 16:22 7bd30558 test/dir/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a
4 Stored 4 0% 2021-07-29 16:22 7bd30558 test/dir/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a
4 Stored 4 0% 2021-07-29 16:22 7bd30558 test/dir/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a
4 Stored 4 0% 2021-07-29 16:22 7bd30558 test/dir/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a/a
-------- ------- --- -------
160 160 0% 42 files
+ cd extract
+ FF_USE_FASTZIP=true
+ gitlab-runner cache-extractor --file ../cache.zip
Runtime platform arch=amd64 os=linux pid=3193878 revision=8925d9a0 version=14.1.0
No URL provided, cache will not be downloaded from shared cache server. Instead a local version of cache will be extracted.
FATAL: remove /home/henrik/runnerrepro/extract/test/dir/a/a: directory not emptyUsed GitLab Runner version
Version: 14.1.0
Git revision: 8925d9a0
Git branch: 14-1-stable
GO version: go1.13.8
Built: 2021-07-20T11:43:26+0000
OS/Arch: linux/amd64Edited by Henrik Karlsson