Fix cache extractor redownloading up-to-date caches for Go Cloud URLs
For pre-signed URLs, the runner helper checks that the Last-Modified
header of the cache blob is later than the modification time of the
downloaded cache file. This avoids having to download an unchanged
cache file.
This was previously not happening for Go Cloud URLs. This commit adds support for this checking.
Note that the Last-Modified time appears to be updated automatically by Go Cloud, so we don't to set this header as we do with presigned URLs in the cache archiver.
Why was this MR needed?
It appears for Azure blobs and S3 configs using UploadARN, the cache would always be redownloaded.
What's the best way to test this MR?
- Set up a runner with Azure Blob Storage and configure
cache_dir:
[[runners]]
name = "jet-arm.local"
url = "https://stanhu-aws.stanhu.com"
id = 3
token = "glrt-xXVaTgQHwsvhyzWzXZfC"
token_obtained_at = 2022-10-27T22:09:33Z
token_expires_at = 0001-01-01T00:00:00Z
executor = "docker"
[runners.feature_flags]
# FF_USE_LEGACY_S3_CACHE_ADAPTER = true
[runners.custom_build_dir]
[runners.cache]
Type = "azure"
MaxUploadedArchiveSize = 0
[runners.cache.azure]
AccountName = "YOUR-ACCOUNT-NAME"
AccountKey = "REDACTED"
ContainerName = "YOUR-BLOB-STORAGE-CONTAINER"
StorageDomain = "blob.core.windows.net"
[runners.docker]
helper_image = "docker.io/library/gitlab-runner-helper:local"
disable_cache = false
volumes = ["/cache"]
shm_size = 0
cache_dir = "/cache"
- Run
rm -f out/binaries/gitlab-runner-* && make helper-bin-linux runner-local-image. - Run
./out/binaries/gitlab-runner-linux-amd64 run -c <your config file> - Then test with a simple
.gitlab-ci.yml:
default:
script:
- echo "hello world" > test.txt
cache:
paths:
- test.txt
artifacts:
paths:
- test.txt
On the first run, you should see the cache downloaded:
Hit retry, and you should see that the file is not redownloaded:
What are the relevant issue numbers?
Edited by Stan Hu

