Kubernetes Runner Container ContainersNotReady
Status update (2023-10-23)
-
In order to provide more info in the build log, we added logging of all the incoming Kubernetes events for the build pod.
-
To enable this logging capability, toggle the
FF_PRINT_POD_EVENTS
feature flag. -
Note: If there are specific issues that need to be investigated or if you wish us to add enhancements to this feature, please add those in new issues as this one has quite a few comments already and filtering through everything would be hard.
Status update (2022-11-07)
After reviewing the comments to date on this issue, we conclude that while the failures seem random at first, it's likely that some specific resource or node is preventing the pods from starting. So investigating the Kubernetes cluster configuration and analyzing the cluster is the first set of troubleshooting steps to take. If you still cannot resolve the issue then follow the steps below to provide us additional information so that we can determine if additional investigation is required specific to the Runner.
Failure analysis steps:
- Inspect the failing pod with
kubectl describe pod <pod>
- Get the logs from all containers:
kubectl logs -c init-permissions kubectl logs -c helper kubectl logs -c build?
- Provide your
config.toml
file, sanitized.
Likely root cause
- A specific resource or node is preventing the pods from starting so that means that the cluster configuration is the problem.
Summary
I'm running a pipeline and witnessing a strange problem with Kubernetes git-lab-runner
The Job started with failing and fetching up the wrong pod at random. The Job's run well sometimes and sometimes it just fetch the wrong pod name
For example the runner should run on my-automation-gitlab-runner-6cbffc6c8-cpvzn
instead it runs on my-automation-gitlab-runner-5484fbc54b-nbp8b
Running with gitlab-runner 14.2.0 (58ba2b95) on my-automation-gitlab-runner-5484fbc54b-nbp8b **ETj*****
Preparing the "kubernetes" executor
Using Kubernetes namespace: gitlab-runners
Using Kubernetes executor with image ubuntu:20.04 ...
8Using attach strategy to execute scripts...
Preparing environment
Waiting for pod gitlab-runners/runner-etjxlzsb-project-3830-concurrent-0ghjh2 to be running, status is Pending
Waiting for pod gitlab-runners/runner-etjxlzsb-project-3830-concurrent-0ghjh2 to be running, status is Pending
ContainersNotInitialized: "containers with incomplete status: [init-permissions]"
ContainersNotReady: "containers with unready status: [build helper]"
ContainersNotReady: "containers with unready status: [build helper]"
Runner: #34721 (**ETj*****) my-automation-gitlab-runner-5484fbc54b-nbp8b```
When i grep the pod the name appear to be `my-automation-gitlab-runner-6cbffc6c8-cpvzn
❯ kubectl get pods -n gitlab-runners | grep my-automation-gitlab-runner
my-automation-gitlab-runner-6cbffc6c8-cpvzn 1/1 Running 0 170m
~ ❯
There is a mismatch in the name coming intermittently