On Kubernetes executor using persistent volumes, jobs start before PVCs are detached from previous pods leading to startup delay

Summary

When using the Kubernetes executor using "Persistent per-concurrency build volumes" as described here, when all runner indices are in use, it is possible that a new job for CI_CONCURRENT_ID n will start before the previous pod for ID n is deleted. The new pod will not be able to attach the PVC until the previous pod has terminated, leading to a delay.

Steps to reproduce

Configure a Kubernetes executor with this config.
Launch a number of CI jobs greater than the concurrency configured for the runner.

Actual behavior

The n+1th job spends a greater amount of time starting, and job logs contain many repeats of:

	ContainersNotReady: "containers with unready status: [build helper svc-0]"
	ContainersNotReady: "containers with unready status: [build helper svc-0]"
Waiting for pod ads-cicd/runner-ys6cxeh6u-project-112-concurrent-0-bwo89sb2 to be running, status is Pending
	ContainersNotInitialized: "containers with incomplete status: [init-permissions]"

If the events for this pod are inspected with `kubectl events , they will contain warnings such as:

12m         Warning   FailedAttachVolume       Pod/runner-ys6cxeh6u-project-112-concurrent-4-r3puy4gr    Multi-Attach error for volume "pvc-cf6dc265-5a4f-46b5-8f46-326aeb4ddd13" Volume is already used by pod(s) runner-ys6cxeh6u-project-112-concurrent-4-f2hqwc3k

Expected behavior

The job will launch with a runner index that has a PVC that is available to attach because the pod it was previously attached to has terminated. If none are available, it will remain in a pending state.

Relevant logs and/or screenshots

job log

Running with gitlab-runner 17.4.2 (24b8e03b)
  on <project>: GRaaS runner for project <project> deployed in ads-cicd namespace in us-east-1 with cluster context <context> Ys6CXEh6U, system ID:  <id>
Resolving secrets
Preparing the "kubernetes" executor
00:00
Using Kubernetes namespace: <namespace>
Using Kubernetes executor with image <image> ...
Using attach strategy to execute scripts...
Preparing environment
00:46
Using FF_USE_POD_ACTIVE_DEADLINE_SECONDS, the Pod activeDeadlineSeconds will be set to the job timeout: 5h0m0s...
Waiting for pod ads-cicd/runner-ys6cxeh6u-project-112-concurrent-0-bwo89sb2 to be running, status is Pending
Waiting for pod ads-cicd/runner-ys6cxeh6u-project-112-concurrent-0-bwo89sb2 to be running, status is Pending
	ContainersNotInitialized: "containers with incomplete status: [init-permissions]"
	ContainersNotReady: "containers with unready status: [build helper svc-0]"
	ContainersNotReady: "containers with unready status: [build helper svc-0]"
Waiting for pod ads-cicd/runner-ys6cxeh6u-project-112-concurrent-0-bwo89sb2 to be running, status is Pending
	ContainersNotInitialized: "containers with incomplete status: [init-permissions]"
	ContainersNotReady: "containers with unready status: [build helper svc-0]"
	ContainersNotReady: "containers with unready status: [build helper svc-0]"
Waiting for pod ads-cicd/runner-ys6cxeh6u-project-112-concurrent-0-bwo89sb2 to be running, status is Pending
	ContainersNotInitialized: "containers with incomplete status: [init-permissions]"
	ContainersNotReady: "containers with unready status: [build helper svc-0]"
	ContainersNotReady: "containers with unready status: [build helper svc-0]"
Waiting for pod ads-cicd/runner-ys6cxeh6u-project-112-concurrent-0-bwo89sb2 to be running, status is Pending
	ContainersNotInitialized: "containers with incomplete status: [init-permissions]"
	ContainersNotReady: "containers with unready status: [build helper svc-0]"
	ContainersNotReady: "containers with unready status: [build helper svc-0]"
Waiting for pod ads-cicd/runner-ys6cxeh6u-project-112-concurrent-0-bwo89sb2 to be running, status is Pending
	ContainersNotInitialized: "containers with incomplete status: [init-permissions]"
	ContainersNotReady: "containers with unready status: [build helper svc-0]"
	ContainersNotReady: "containers with unready status: [build helper svc-0]"
Waiting for pod ads-cicd/runner-ys6cxeh6u-project-112-concurrent-0-bwo89sb2 to be running, status is Pending
	ContainersNotInitialized: "containers with incomplete status: [init-permissions]"
	ContainersNotReady: "containers with unready status: [build helper svc-0]"
	ContainersNotReady: "containers with unready status: [build helper svc-0]"
Waiting for pod ads-cicd/runner-ys6cxeh6u-project-112-concurrent-0-bwo89sb2 to be running, status is Pending
	ContainersNotInitialized: "containers with incomplete status: [init-permissions]"
	ContainersNotReady: "containers with unready status: [build helper svc-0]"
	ContainersNotReady: "containers with unready status: [build helper svc-0]"
Waiting for pod ads-cicd/runner-ys6cxeh6u-project-112-concurrent-0-bwo89sb2 to be running, status is Pending
	ContainersNotInitialized: "containers with incomplete status: [init-permissions]"
	ContainersNotReady: "containers with unready status: [build helper svc-0]"
	ContainersNotReady: "containers with unready status: [build helper svc-0]"
Waiting for pod ads-cicd/runner-ys6cxeh6u-project-112-concurrent-0-bwo89sb2 to be running, status is Pending
	ContainersNotInitialized: "containers with incomplete status: [init-permissions]"
	ContainersNotReady: "containers with unready status: [build helper svc-0]"
	ContainersNotReady: "containers with unready status: [build helper svc-0]"
Waiting for pod ads-cicd/runner-ys6cxeh6u-project-112-concurrent-0-bwo89sb2 to be running, status is Pending
	ContainersNotInitialized: "containers with incomplete status: [init-permissions]"
	ContainersNotReady: "containers with unready status: [build helper svc-0]"
	ContainersNotReady: "containers with unready status: [build helper svc-0]"
Waiting for pod ads-cicd/runner-ys6cxeh6u-project-112-concurrent-0-bwo89sb2 to be running, status is Pending
	ContainersNotInitialized: "containers with incomplete status: [init-permissions]"
	ContainersNotReady: "containers with unready status: [build helper svc-0]"
	ContainersNotReady: "containers with unready status: [build helper svc-0]"
Waiting for pod ads-cicd/runner-ys6cxeh6u-project-112-concurrent-0-bwo89sb2 to be running, status is Pending
	ContainersNotInitialized: "containers with incomplete status: [init-permissions]"
	ContainersNotReady: "containers with unready status: [build helper svc-0]"
	ContainersNotReady: "containers with unready status: [build helper svc-0]"
Waiting for pod ads-cicd/runner-ys6cxeh6u-project-112-concurrent-0-bwo89sb2 to be running, status is Pending
	ContainersNotInitialized: "containers with incomplete status: [init-permissions]"
	ContainersNotReady: "containers with unready status: [build helper svc-0]"
	ContainersNotReady: "containers with unready status: [build helper svc-0]"
Waiting for pod ads-cicd/runner-ys6cxeh6u-project-112-concurrent-0-bwo89sb2 to be running, status is Pending
	ContainersNotInitialized: "containers with incomplete status: [init-permissions]"
	ContainersNotReady: "containers with unready status: [build helper svc-0]"
	ContainersNotReady: "containers with unready status: [build helper svc-0]"
Running on runner-ys6cxeh6u-project-112-concurrent-0-bwo89sb2 via project-112-cosmos-bazel-gitlab-runner-5574bcfd46-88l5j...

Environment description

Enterprise edition, Kubernetes executor

config.toml contents

Add your configuration here

Used GitLab Runner version

Running with gitlab-runner 17.4.2 (24b8e03b)