Kubernetes executor failing randomly to start jobs
Summary
Our builds are randomly failing with the error:
ERROR: Job failed (system failure): prepare environment: timed out waiting for pod to start. Check https://docs.gitlab.com/runner/shells/index.html#shell-profile-loading for more information
We use GKE for our gitlab builds cluster, nothing fancy and are running version 13.3.2 of the runner docker image, and our GitLab enterprise edition version is 13.3.6-ee There are tonnes of resource in the cluster, so it's not that the runner manager can't get resources.. Any ideas?
Steps to reproduce
Hard to give a concrete set of steps, we have a dozens of separate builds that run within Gitlab
Example Project
What is the current bug behavior?
Builds timeout
What is the expected correct behavior?
Builds run
Relevant logs and/or screenshots
[0KRunning with gitlab-runner 13.3.2 (4df93c99)
[0;m[0K on cloud-gitlab-runner-gitlab-runner-55b5c95b4-24nk7 GwR2cTCr
[0;msection_start:1602265773:prepare_executor
[0K[0K[36;1mPreparing the "kubernetes" executor[0;m
[0;m[0KUsing Kubernetes namespace: gitlab-managed-apps
[0;m[0KUsing Kubernetes executor with image quay.io/helmpack/chart-testing:v3.0.0-rc.1 ...
[0;msection_end:1602265773:prepare_executor
[0Ksection_start:1602265773:prepare_script
[0K[0K[36;1mPreparing environment[0;m
[0;mWaiting for pod gitlab-managed-apps/runner-gwr2ctcr-project-5990-concurrent-04f2s8 to be running, status is Pending
Waiting for pod gitlab-managed-apps/runner-gwr2ctcr-project-5990-concurrent-04f2s8 to be running, status is Pending
Waiting for pod gitlab-managed-apps/runner-gwr2ctcr-project-5990-concurrent-04f2s8 to be running, status is Pending
Waiting for pod gitlab-managed-apps/runner-gwr2ctcr-project-5990-concurrent-04f2s8 to be running, status is Pending
Waiting for pod gitlab-managed-apps/runner-gwr2ctcr-project-5990-concurrent-04f2s8 to be running, status is Pending
Waiting for pod gitlab-managed-apps/runner-gwr2ctcr-project-5990-concurrent-04f2s8 to be running, status is Pending
Waiting for pod gitlab-managed-apps/runner-gwr2ctcr-project-5990-concurrent-04f2s8 to be running, status is Pending
Waiting for pod gitlab-managed-apps/runner-gwr2ctcr-project-5990-concurrent-04f2s8 to be running, status is Pending
Waiting for pod gitlab-managed-apps/runner-gwr2ctcr-project-5990-concurrent-04f2s8 to be running, status is Pending
Waiting for pod gitlab-managed-apps/runner-gwr2ctcr-project-5990-concurrent-04f2s8 to be running, status is Pending
Waiting for pod gitlab-managed-apps/runner-gwr2ctcr-project-5990-concurrent-04f2s8 to be running, status is Pending
Waiting for pod gitlab-managed-apps/runner-gwr2ctcr-project-5990-concurrent-04f2s8 to be running, status is Pending
Waiting for pod gitlab-managed-apps/runner-gwr2ctcr-project-5990-concurrent-04f2s8 to be running, status is Pending
Waiting for pod gitlab-managed-apps/runner-gwr2ctcr-project-5990-concurrent-04f2s8 to be running, status is Pending
Waiting for pod gitlab-managed-apps/runner-gwr2ctcr-project-5990-concurrent-04f2s8 to be running, status is Pending
Waiting for pod gitlab-managed-apps/runner-gwr2ctcr-project-5990-concurrent-04f2s8 to be running, status is Pending
Waiting for pod gitlab-managed-apps/runner-gwr2ctcr-project-5990-concurrent-04f2s8 to be running, status is Pending
Waiting for pod gitlab-managed-apps/runner-gwr2ctcr-project-5990-concurrent-04f2s8 to be running, status is Pending
Waiting for pod gitlab-managed-apps/runner-gwr2ctcr-project-5990-concurrent-04f2s8 to be running, status is Pending
Waiting for pod gitlab-managed-apps/runner-gwr2ctcr-project-5990-concurrent-04f2s8 to be running, status is Pending
Waiting for pod gitlab-managed-apps/runner-gwr2ctcr-project-5990-concurrent-04f2s8 to be running, status is Pending
Waiting for pod gitlab-managed-apps/runner-gwr2ctcr-project-5990-concurrent-04f2s8 to be running, status is Pending
Waiting for pod gitlab-managed-apps/runner-gwr2ctcr-project-5990-concurrent-04f2s8 to be running, status is Pending
Waiting for pod gitlab-managed-apps/runner-gwr2ctcr-project-5990-concurrent-04f2s8 to be running, status is Pending
Waiting for pod gitlab-managed-apps/runner-gwr2ctcr-project-5990-concurrent-04f2s8 to be running, status is Pending
Waiting for pod gitlab-managed-apps/runner-gwr2ctcr-project-5990-concurrent-04f2s8 to be running, status is Pending
Waiting for pod gitlab-managed-apps/runner-gwr2ctcr-project-5990-concurrent-04f2s8 to be running, status is Pending
Waiting for pod gitlab-managed-apps/runner-gwr2ctcr-project-5990-concurrent-04f2s8 to be running, status is Pending
Waiting for pod gitlab-managed-apps/runner-gwr2ctcr-project-5990-concurrent-04f2s8 to be running, status is Pending
Waiting for pod gitlab-managed-apps/runner-gwr2ctcr-project-5990-concurrent-04f2s8 to be running, status is Pending
Waiting for pod gitlab-managed-apps/runner-gwr2ctcr-project-5990-concurrent-04f2s8 to be running, status is Pending
Waiting for pod gitlab-managed-apps/runner-gwr2ctcr-project-5990-concurrent-04f2s8 to be running, status is Pending
Waiting for pod gitlab-managed-apps/runner-gwr2ctcr-project-5990-concurrent-04f2s8 to be running, status is Pending
Waiting for pod gitlab-managed-apps/runner-gwr2ctcr-project-5990-concurrent-04f2s8 to be running, status is Pending
Waiting for pod gitlab-managed-apps/runner-gwr2ctcr-project-5990-concurrent-04f2s8 to be running, status is Pending
Waiting for pod gitlab-managed-apps/runner-gwr2ctcr-project-5990-concurrent-04f2s8 to be running, status is Pending
Waiting for pod gitlab-managed-apps/runner-gwr2ctcr-project-5990-concurrent-04f2s8 to be running, status is Pending
Waiting for pod gitlab-managed-apps/runner-gwr2ctcr-project-5990-concurrent-04f2s8 to be running, status is Pending
Waiting for pod gitlab-managed-apps/runner-gwr2ctcr-project-5990-concurrent-04f2s8 to be running, status is Pending
Waiting for pod gitlab-managed-apps/runner-gwr2ctcr-project-5990-concurrent-04f2s8 to be running, status is Pending
Waiting for pod gitlab-managed-apps/runner-gwr2ctcr-project-5990-concurrent-04f2s8 to be running, status is Pending
Waiting for pod gitlab-managed-apps/runner-gwr2ctcr-project-5990-concurrent-04f2s8 to be running, status is Pending
Waiting for pod gitlab-managed-apps/runner-gwr2ctcr-project-5990-concurrent-04f2s8 to be running, status is Pending
Waiting for pod gitlab-managed-apps/runner-gwr2ctcr-project-5990-concurrent-04f2s8 to be running, status is Pending
Waiting for pod gitlab-managed-apps/runner-gwr2ctcr-project-5990-concurrent-04f2s8 to be running, status is Pending
Waiting for pod gitlab-managed-apps/runner-gwr2ctcr-project-5990-concurrent-04f2s8 to be running, status is Pending
Waiting for pod gitlab-managed-apps/runner-gwr2ctcr-project-5990-concurrent-04f2s8 to be running, status is Pending
Waiting for pod gitlab-managed-apps/runner-gwr2ctcr-project-5990-concurrent-04f2s8 to be running, status is Pending
Waiting for pod gitlab-managed-apps/runner-gwr2ctcr-project-5990-concurrent-04f2s8 to be running, status is Pending
Waiting for pod gitlab-managed-apps/runner-gwr2ctcr-project-5990-concurrent-04f2s8 to be running, status is Pending
Waiting for pod gitlab-managed-apps/runner-gwr2ctcr-project-5990-concurrent-04f2s8 to be running, status is Pending
Waiting for pod gitlab-managed-apps/runner-gwr2ctcr-project-5990-concurrent-04f2s8 to be running, status is Pending
Waiting for pod gitlab-managed-apps/runner-gwr2ctcr-project-5990-concurrent-04f2s8 to be running, status is Pending
Waiting for pod gitlab-managed-apps/runner-gwr2ctcr-project-5990-concurrent-04f2s8 to be running, status is Pending
Waiting for pod gitlab-managed-apps/runner-gwr2ctcr-project-5990-concurrent-04f2s8 to be running, status is Pending
Waiting for pod gitlab-managed-apps/runner-gwr2ctcr-project-5990-concurrent-04f2s8 to be running, status is Pending
Waiting for pod gitlab-managed-apps/runner-gwr2ctcr-project-5990-concurrent-04f2s8 to be running, status is Pending
Waiting for pod gitlab-managed-apps/runner-gwr2ctcr-project-5990-concurrent-04f2s8 to be running, status is Pending
Waiting for pod gitlab-managed-apps/runner-gwr2ctcr-project-5990-concurrent-04f2s8 to be running, status is Pending
Waiting for pod gitlab-managed-apps/runner-gwr2ctcr-project-5990-concurrent-04f2s8 to be running, status is Pending
Waiting for pod gitlab-managed-apps/runner-gwr2ctcr-project-5990-concurrent-04f2s8 to be running, status is Pending
section_end:1602265957:prepare_script
[0K[31;1mERROR: Job failed (system failure): prepare environment: timed out waiting for pod to start. Check https://docs.gitlab.com/runner/shells/index.html#shell-profile-loading for more information
[0;m
Output of checks
Results of GitLab environment info
Expand for output related to GitLab environment info
(For installations with omnibus-gitlab package run and paste the output of: `sudo gitlab-rake gitlab:env:info`) (For installations from source run and paste the output of: `sudo -u git -H bundle exec rake gitlab:env:info RAILS_ENV=production`)
Results of GitLab application Check
Expand for output related to the GitLab application check
(For installations with omnibus-gitlab package run and paste the output of:
sudo gitlab-rake gitlab:check SANITIZE=true
)(For installations from source run and paste the output of:
sudo -u git -H bundle exec rake gitlab:check RAILS_ENV=production SANITIZE=true
)(we will only investigate if the tests are passing)