Create folder when doing a curl to download vendor/gems
Context
Follow-up of !127104 (merged).
The problem
We're trying to download the vendor/gems
folder from the API in a job that didn't do a git clone
(i.e. there are no folders in the working directory). We're making it an archive of everything inside vendor/gems
, and we store it in vendor/gems.tar.gz
:
for folder_path in "vendor/gems" "gems"; do
local output="${folder_path}.tar.gz"
If we didn't download the gems cache beforehand, the vendor
folder will not be present, and it won't be created. curl cannot then write to the vendor
folder and errors out.
Why did it only occur in forks?
In the failing jobs, we didn't download a gem cache (example), so the vendor
folder wasn't there.
What does this MR do and why?
Add the --create-dirs
option to curl
to respect the folder hierarchy.
We could have created the folder manually with mkdir vendor
, but it feels less brittle to create any folder hierarchy we're trying to download.
Steps to reproduce locally
pre-requisites
docker run -it ruby:3.0-alpine3.16 ash
apk add curl
mkdir /app
cd app
export CI_API_V4_URL="https://gitlab.com/api/v4"
export CI_PROJECT_ID="48263887"
export CI_COMMIT_SHA="7578f0270c42cb0940437de741c1072ef20eebd0"
export CI_PROJECT_VISIBILITY="public"
Before
# Copy/paste the old download_local_gems function from scripts/utils.sh
$ download_local_gems
Downloading vendor/gems
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0Warning: Failed to open the file vendor/gems.tar.gz: No such file or directory
100 10934 0 10934 0 0 6343 0 --:--:-- 0:00:01 --:--:-- 6342
curl: (23) Failure writing output to destination
tar: can't open 'vendor/gems.tar.gz': No such file or directory
rm: can't remove 'vendor/gems.tar.gz': No such file or directory
Downloading gems
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 123k 0 123k 0 0 74163 0 --:--:-- 0:00:01 --:--:-- 74187
After
# Copy/paste the new download_local_gems function from scripts/utils.sh
$ download_local_gems
Downloading vendor/gems
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 141k 0 141k 0 0 82539 0 --:--:-- 0:00:01 --:--:-- 82531
Downloading gems
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 123k 0 123k 0 0 70828 0 --:--:-- 0:00:01 --:--:-- 70823
$ ls -l
total 8
drwxrwxr-x 13 root root 4096 Aug 7 07:33 gems
drwxrwxr-x 3 root root 4096 Aug 7 07:33 vendor
MR acceptance checklist
This checklist encourages us to confirm any changes have been analyzed to reduce risks in quality, performance, reliability, security, and maintainability.
-
I have evaluated the MR acceptance checklist for this MR.