Skip to content

Kubernetes Runner Container ContainersNotReady

Status update (2023-10-23)

  • In order to provide more info in the build log, we added logging of all the incoming Kubernetes events for the build pod.

  • To enable this logging capability, toggle the FF_PRINT_POD_EVENTS feature flag.

  • Note: If there are specific issues that need to be investigated or if you wish us to add enhancements to this feature, please add those in new issues as this one has quite a few comments already and filtering through everything would be hard.

Status update (2022-11-07)

After reviewing the comments to date on this issue, we conclude that while the failures seem random at first, it's likely that some specific resource or node is preventing the pods from starting. So investigating the Kubernetes cluster configuration and analyzing the cluster is the first set of troubleshooting steps to take. If you still cannot resolve the issue then follow the steps below to provide us additional information so that we can determine if additional investigation is required specific to the Runner.

Failure analysis steps:

  1. Inspect the failing pod with kubectl describe pod <pod>
  2. Get the logs from all containers: kubectl logs -c init-permissions kubectl logs -c helper kubectl logs -c build?
  3. Provide your config.toml file, sanitized.

Likely root cause

  • A specific resource or node is preventing the pods from starting so that means that the cluster configuration is the problem.

Summary

I'm running a pipeline and witnessing a strange problem with Kubernetes git-lab-runner

The Job started with failing and fetching up the wrong pod at random. The Job's run well sometimes and sometimes it just fetch the wrong pod name

For example the runner should run on my-automation-gitlab-runner-6cbffc6c8-cpvzn instead it runs on my-automation-gitlab-runner-5484fbc54b-nbp8b

Running with gitlab-runner 14.2.0 (58ba2b95) on my-automation-gitlab-runner-5484fbc54b-nbp8b **ETj*****
Preparing the "kubernetes" executor
Using Kubernetes namespace: gitlab-runners
Using Kubernetes executor with image ubuntu:20.04 ...
8Using attach strategy to execute scripts...

Preparing environment
Waiting for pod gitlab-runners/runner-etjxlzsb-project-3830-concurrent-0ghjh2 to be running, status is Pending
Waiting for pod gitlab-runners/runner-etjxlzsb-project-3830-concurrent-0ghjh2 to be running, status is Pending
	ContainersNotInitialized: "containers with incomplete status: [init-permissions]"
	ContainersNotReady: "containers with unready status: [build helper]"
	ContainersNotReady: "containers with unready status: [build helper]"
Runner: #34721 (**ETj*****) my-automation-gitlab-runner-5484fbc54b-nbp8b```

When i grep the pod the name appear to be `my-automation-gitlab-runner-6cbffc6c8-cpvzn

❯ kubectl get pods -n  gitlab-runners | grep my-automation-gitlab-runner
my-automation-gitlab-runner-6cbffc6c8-cpvzn           1/1     Running   0          170m
~ ❯

There is a mismatch in the name coming intermittently

Edited by Darren Eastman