Fix cache extractor redownloading up-to-date caches for Go Cloud URLs

For pre-signed URLs, the runner helper checks that the Last-Modified header of the cache blob is later than the modification time of the downloaded cache file. This avoids having to download an unchanged cache file.

This was previously not happening for Go Cloud URLs. This commit adds support for this checking.

Note that the Last-Modified time appears to be updated automatically by Go Cloud, so we don't to set this header as we do with presigned URLs in the cache archiver.

Why was this MR needed?

It appears for Azure blobs and S3 configs using UploadARN, the cache would always be redownloaded.

What's the best way to test this MR?

  1. Set up a runner with Azure Blob Storage and configure cache_dir:
[[runners]]
  name = "jet-arm.local"
  url = "https://stanhu-aws.stanhu.com"
  id = 3
  token = "glrt-xXVaTgQHwsvhyzWzXZfC"
  token_obtained_at = 2022-10-27T22:09:33Z
  token_expires_at = 0001-01-01T00:00:00Z
  executor = "docker"
  [runners.feature_flags]
  # FF_USE_LEGACY_S3_CACHE_ADAPTER = true
  [runners.custom_build_dir]
  [runners.cache]
    Type = "azure"
    MaxUploadedArchiveSize = 0
  [runners.cache.azure]
    AccountName = "YOUR-ACCOUNT-NAME"
    AccountKey = "REDACTED"
    ContainerName = "YOUR-BLOB-STORAGE-CONTAINER"
    StorageDomain = "blob.core.windows.net"
  [runners.docker]
    helper_image = "docker.io/library/gitlab-runner-helper:local"
    disable_cache = false
    volumes = ["/cache"]
    shm_size = 0
    cache_dir = "/cache"
  1. Run rm -f out/binaries/gitlab-runner-* && make helper-bin-linux runner-local-image.
  2. Run ./out/binaries/gitlab-runner-linux-amd64 run -c <your config file>
  3. Then test with a simple .gitlab-ci.yml:
default:
  script:
    - echo "hello world" > test.txt
  cache:
    paths:
      - test.txt
  artifacts:
    paths:
      - test.txt

On the first run, you should see the cache downloaded:

image

Hit retry, and you should see that the file is not redownloaded:

image

What are the relevant issue numbers?

Edited by Stan Hu

Merge request reports

Loading