Pipeline hangs on cache operations
## Summary
When restoring or updating the S3 cache, our pipeline hangs for a minute at each step.
## Steps to reproduce
Configure the cache at the runner level:
<details>
<summary>values.yaml</summary>
```yaml
checkInterval: 10
concurrent: 10
extraEnv:
RUNNER_OUTPUT_LIMIT: '128000'
image:
image: gitlab-org/gitlab-runner
registry: registry.gitlab.com
imagePullPolicy: Always
logLevel: debug
rbac:
create: true
serviceAccount:
create: true
runners:
cache: {}
config: |-
[[runners]]
environment = ["RUNNER_OUTPUT_LIMIT=128000"]
[runners.kubernetes]
cpu_request = "1000m"
cpu_limit = "8000m"
memory_request = "2000Mi"
memory_limit = "8000Mi"
helper_cpu_request = "100m"
helper_cpu_limit = "1000m"
helper_memory_request = "128Mi"
helper_memory_limit = "512Mi"
service_cpu_request = "1000m"
service_cpu_limit = "2000m"
service_memory_request = "512Mi"
service_memory_limit = "1024Mi"
helper_image = "oci.repo.mycorp.de/integration/gitlab-runner-helper:18.9.0"
pull_policy = ["if-not-present"]
privileged = false
pod_annotations_overwrite_allowed = "container.apparmor.security.beta.kubernetes.io/.*"
[[runners.kubernetes.volumes.secret]]
name = "gitlab-runner-mycorp-ca"
mount_path = "/etc/gitlab-runner/certs/"
[runners.cache]
Type = "s3"
Shared = true
[runners.cache.s3]
ServerAddress = "ecs.internal.mycorp.de:9021"
AccessKey = "gitlab-runner"
SecretKey = "XXX"
BucketName = "gitlab-runner"
runUntagged: false
sessionServer:
enabled: false
unregisterRunners: true
certsSecretName: gitlab-runner-mycorp-ca
gitlabUrl: https://gitlab.internal.mycorp.de
runnerToken: XXX
resources:
requests:
memory: 128Mi
cpu: 200m
limits:
memory: 256Mi
cpu: 1000m
```
</details>
Then run a job using the cache:
<!--
Please add the definition of the job from `.gitlab-ci.yml` that is failing
inside of the code blocks (```) below.
-->
<details>
<summary> .gitlab-ci.yml </summary>
```yml
build:
image: alpine
script:
- mkdir -p cache
- echo "Hello World" > cache/hello
cache:
paths:
- ./cache
key: cache
```
</details>
## Actual behavior
Our pipeline is stuck for a minute at restoring the cache, regardless of whether it exists or not. It's also stuck at storing the cache, also for a minute.
## Expected behavior
I'd expect that cache uploading and downloading doesn't get stuck.
## Relevant logs and/or screenshots
<!--
Paste the job logs inside of the code blocks (```) below so it would be
easier to read.
-->
<details>
<summary> job log </summary>
```sh
2026-02-27T08:14:24.270672Z 00O [0KRunning with gitlab-runner 18.9.0 (07e534ba)[0;m
2026-02-27T08:14:24.270704Z 00O [0K on gitlab-runner-6978dd4ffd-f2mxw Zzoyc2cKb, system ID: r_cr7u3Z5ams79[0;m
2026-02-27T08:14:24.270741Z 00O section_start:1772180064:prepare_executor
2026-02-27T08:14:24.270742Z 00O+[0K[0K[36;1mPreparing the "kubernetes" executor[0;m[0;m
2026-02-27T08:14:24.270970Z 00O [0KUsing Kubernetes namespace: scm-test[0;m
2026-02-27T08:14:24.271245Z 00O [0KUsing Kubernetes executor with image oci.repo.mycorp.de/integration/ubuntu:minimal ...[0;m
2026-02-27T08:14:24.271262Z 00O [0KUsing attach strategy to execute scripts...[0;m
2026-02-27T08:14:24.271278Z 00O [0KUsing effective pull policy of [IfNotPresent] for container build[0;m
2026-02-27T08:14:24.271292Z 00O [0KUsing effective pull policy of [IfNotPresent] for container helper[0;m
2026-02-27T08:14:24.271304Z 00O [0KUsing effective pull policy of [IfNotPresent] for container init-permissions[0;m
2026-02-27T08:14:24.375923Z 00O section_end:1772180064:prepare_executor
2026-02-27T08:14:24.375932Z 00O+[0Ksection_start:1772180064:prepare_script
2026-02-27T08:14:24.376172Z 00O+[0K[0K[36;1mPreparing environment[0;m[0;m
2026-02-27T08:14:24.381031Z 00O [0KUsing FF_USE_POD_ACTIVE_DEADLINE_SECONDS, the Pod activeDeadlineSeconds will be set to the job timeout: 1h0m0s...[0;m
2026-02-27T08:14:24.402887Z 00E Waiting for pod scm-test/runner-zzoyc2ckb-project-3455-concurrent-0-4w0w0f1i to be running, status is Pending
2026-02-27T08:14:27.589184Z 01O Running on runner-zzoyc2ckb-project-3455-concurrent-0-4w0w0f1i via gitlab-runner-6978dd4ffd-f2mxw...
2026-02-27T08:14:27.589226Z 01O
2026-02-27T08:14:27.589328Z 00O section_end:1772180067:prepare_script
2026-02-27T08:14:27.589331Z 00O+[0Ksection_start:1772180067:get_sources
2026-02-27T08:14:27.589729Z 00O+[0K[0K[36;1mGetting source from Git repository[0;m[0;m
2026-02-27T08:14:28.089593Z 01O [32;1mGitaly correlation ID: 01KJF2KHPP4WSF4PT48JTSKMWK[0;m
2026-02-27T08:14:28.089612Z 01O [32;1mFetching changes with git depth set to 20...[0;m
2026-02-27T08:14:28.089618Z 01O Initialized empty Git repository in /builds/mydept-infra/playground/ci-cache-test/.git/
2026-02-27T08:14:28.089621Z 01O [32;1mCreated fresh repository.[0;m
2026-02-27T08:14:28.089624Z 01O [32;1mChecking out f423e8f5 as detached HEAD (ref is user-main-patch-0c3b)...[0;m
2026-02-27T08:14:28.590193Z 01O
2026-02-27T08:14:28.590208Z 01O [32;1mSkipping Git submodules setup[0;m
2026-02-27T08:14:28.590213Z 01O
2026-02-27T08:14:28.590325Z 00O section_end:1772180068:get_sources
2026-02-27T08:14:28.590327Z 00O+[0Ksection_start:1772180128:restore_cache
2026-02-27T08:15:28.592453Z 00O+[0K[0K[36;1mRestoring cache[0;m[0;m
2026-02-27T08:15:29.162320Z 01O [32;1mChecking cache for cache-1-protected...[0;m
2026-02-27T08:15:29.162576Z 01O [0;33mWARNING: file does not exist [0;m
2026-02-27T08:15:29.162592Z 01O [0;33mFailed to extract cache[0;m
2026-02-27T08:15:29.162594Z 01O
2026-02-27T08:15:29.162683Z 00O section_end:1772180129:restore_cache
2026-02-27T08:15:29.162684Z 00O+[0Ksection_start:1772180129:step_script
2026-02-27T08:15:29.163607Z 00O+[0K[0K[36;1mExecuting "step_script" stage of the job script[0;m[0;m
2026-02-27T08:15:29.662392Z 01O [32;1m$ mkdir -p cache[0;m
2026-02-27T08:15:29.662420Z 01O [32;1m$ echo "Hello World" > cache/hello[0;m
2026-02-27T08:15:29.662426Z 01O
2026-02-27T08:15:29.662481Z 00O section_end:1772180129:step_script
2026-02-27T08:15:29.662482Z 00O+[0Ksection_start:1772180189:archive_cache
2026-02-27T08:16:29.664360Z 00O+[0K[0K[36;1mSaving cache for successful job[0;m[0;m
2026-02-27T08:16:30.229178Z 01O [32;1mCreating cache cache-1-protected...[0;m
2026-02-27T08:16:30.229188Z 01O ./cache: found 2 matching artifact files and directories[0;m
2026-02-27T08:16:30.229193Z 01O Uploading cache.zip to https://ecs.internal.mycorp.de:9021/gitlab-runner/project/3455/cache-1-protected[0;m
2026-02-27T08:16:30.229198Z 01O [32;1mCreated cache[0;m
2026-02-27T08:16:30.229201Z 01O
2026-02-27T08:16:30.229256Z 00O section_end:1772180190:archive_cache
2026-02-27T08:16:30.229257Z 00O+[0Ksection_start:1772180190:cleanup_file_variables
2026-02-27T08:16:30.229642Z 00O+[0K[0K[36;1mCleaning up project directory and file based variables[0;m[0;m
2026-02-27T08:16:30.729750Z 01O
2026-02-27T08:16:30.729891Z 00O section_end:1772180190:cleanup_file_variables
2026-02-27T08:16:30.729892Z 00O+[0K
2026-02-27T08:16:30.737067Z 00O [32;1mJob succeeded[0;m
```
</details>
## Environment description
We are using GitLab CE 18.9.0, and GitLab Runner Helm Chart 0.86.0. GitLab runs in a virtual machine on Ubuntu 24.04, the Runner is deployed in a Rancher RKE2 cluster. The S3 storage is provided by a Dell ECS.
<!--
Please add the contents of `config.toml` inside of the code blocks (```)
below, remember to remove any secret tokens!
-->
<details>
<summary>values.yaml</summary>
```yaml
checkInterval: 10
concurrent: 10
extraEnv:
RUNNER_OUTPUT_LIMIT: '128000'
image:
image: gitlab-org/gitlab-runner
registry: registry.gitlab.com
imagePullPolicy: Always
logLevel: debug
rbac:
create: true
serviceAccount:
create: true
runners:
cache: {}
config: |-
[[runners]]
environment = ["RUNNER_OUTPUT_LIMIT=128000"]
[runners.kubernetes]
cpu_request = "1000m"
cpu_limit = "8000m"
memory_request = "2000Mi"
memory_limit = "8000Mi"
helper_cpu_request = "100m"
helper_cpu_limit = "1000m"
helper_memory_request = "128Mi"
helper_memory_limit = "512Mi"
service_cpu_request = "1000m"
service_cpu_limit = "2000m"
service_memory_request = "512Mi"
service_memory_limit = "1024Mi"
helper_image = "oci.repo.mycorp.de/integration/gitlab-runner-helper:18.9.0"
pull_policy = ["if-not-present"]
privileged = false
pod_annotations_overwrite_allowed = "container.apparmor.security.beta.kubernetes.io/.*"
[[runners.kubernetes.volumes.secret]]
name = "gitlab-runner-mycorp-ca"
mount_path = "/etc/gitlab-runner/certs/"
[runners.cache]
Type = "s3"
Shared = true
[runners.cache.s3]
ServerAddress = "ecs.internal.mycorp.de:9021"
AccessKey = "gitlab-runner"
SecretKey = "XXX"
BucketName = "gitlab-runner"
runUntagged: false
sessionServer:
enabled: false
unregisterRunners: true
certsSecretName: gitlab-runner-mycorp-ca
gitlabUrl: https://gitlab.internal.mycorp.de
runnerToken: XXX
resources:
requests:
memory: 128Mi
cpu: 200m
limits:
memory: 256Mi
cpu: 1000m
```
</details>
### Used GitLab Runner version
<!--
Please run and paste the output of `gitlab-runner --version`. If you are using
a Runner where you don't have access to, please paste at least the first lines
the from build log, like:
-->
```
Version: 18.9.0
Git revision: 07e534ba
Git branch: 18-9-stable
GO version: go1.25.7 X:cacheprog
Built: 2026-02-19T16:58:00Z
OS/Arch: linux/amd64
```
## Possible fixes
<!--
(If you can, link to the line of code that might be responsible for the problem)
--->
issue