## Summary When using auto-mounted service account tokens, a Kubernetes runner (on GKE) cannot be shutdown cleanly if the jobs it is watching take over an hour to complete ## Steps to reproduce 1. Create a kubernetes runner on GKE using the GitLab helm chart. Do not modify the `automountServiceAccountToken` field, allowing the default of `true` to be set. 2. Create a pipeline with a job that will use this runner. Ensure the job will take over an hour to complete. 3. Start the pipeline 4. When the pipeline gets to the long-running job, terminate the runner pod on the kubernetes cluster. This should put the runner in to a graceful shutdown state. 5. Wait for 3607 seconds to pass. 6. View the job log on GitLab. You should observe the log is full of `Unauthorised API request` type messages. ## Actual behavior The service account token has expired and so the runner is no longer able to contact the kubernetes API to get the status of the pods it was watching. ## Expected behavior The service account token remains valid throughout the entire shutdown cycle so job information is fed back to GitLab until they complete. When all jobs that the runner is watching are complete, the runner completes its graceful shutdown and terminates. ## Environment description Custom installation of GitLab runners on Google Kubernetes Engine. ### Used GitLab Runner version 17.7.0 ## Possible fixes This is strictly an issue with Kubernetes, being tracked at https://github.com/kubernetes/kubernetes/issues/116481. However, it has been open for 2 years with no signs of being worked on. I think it could be fixed by modifying the following file/line (pseudo code) https://gitlab.com/gitlab-org/gitlab-runner/-/blob/main/executors/kubernetes/util.go?ref_type=heads#L95 ```go if inClusterCfg, err := inClusterConfig(); err == nil { if os.GetEnvironmentVar("GITLAB_KUBERNETES_SERVICE_ACCOUNT_TOKEN_PATH") is set { inClusterCfg.BearerTokenFile = os.GetEnvironmentVar("GITLAB_KUBERNETES_SERVICE_ACCOUNT_TOKEN_PATH") } return inClusterCfg, nil } ```