Kubernetes executor fails to pull image from internal docker registry when job is based on it
Summary
After exactly 1 month of usage gitlab-runner started failing with ERROR: Job failed: image pull failed: Back-off pulling image "registry.***.com/gitlab-ci/golang-ci:latest"
when the image present.
Steps to reproduce
Install GitLab in a Kubernetes Cluster with Helm and use for a month :)
Configuration used
values.yaml
global:
hosts:
domain: ***
externalIP: ***
appConfig:
omniauth:
enabled: true
blockAutoCreatedUsers: false
autoLinkSamlUser: true
allowSingleSignOn: ['saml']
syncProfileFromProvider: ['saml']
providers:
- secret: ssoprovider
defaultProjectsFeatures:
issues: false
wiki: false
snippets: false
gitlab-runner:
runners:
privileged: true
envVars:
- name: RUNNER_BUILDS_DIR
value: /builds
.gitlab-ci.yml example snippet
.golang-lint:
image: registry.***.com/gitlab-ci/golang-ci:latest
script:
- go vet ./...
Current behavior
Kubernetes Executor fails to run jobs based on internal images.
Expected behavior
Kubernetes Executor not failing to run jobs based on internal images.
Versions
- Chart: v1.6.3
- Platform:
- Cloud: GKE
- Kubernetes:
- Client: version.Info{Major:"1", Minor:"13", GitVersion:"v1.13.4", GitCommit:"c27b913fddd1a6c480c229191a087698aa92f0b1", GitTreeState:"clean", BuildDate:"2019-02-28T13:37:52Z", GoVersion:"go1.11.5", Compiler:"gc", Platform:"linux/amd64"}
- Server: version.Info{Major:"1", Minor:"11+", GitVersion:"v1.11.7-gke.12", GitCommit:"06f08e60069231bd21bdf673cf0595aac80b99f6", GitTreeState:"clean", BuildDate:"2019-02-25T20:37:10Z", GoVersion:"go1.10.8b4", Compiler:"gc", Platform:"linux/amd64"}
- Helm:
- Client: &version.Version{SemVer:"v2.13.1", GitCommit:"618447cbf203d147601b4b9bd7f8c37a5d39fbb4", GitTreeState:"clean"}
- Server: &version.Version{SemVer:"v2.13.0", GitCommit:"79d07943b03aea2b76c12644b4b54733bc5958d6", GitTreeState:"clean"}
Relevant logs
Jobs fail in several ways:
Running with gitlab-runner 11.8.0 (4745a6f3)
on gitlab-gitlab-runner-59b4dd5899-tzk7g 6Fcciz8x
Using Kubernetes namespace: gitlab
Using Kubernetes executor with image registry.***.com/gitlab-ci/golang-ci:latest ...
Waiting for pod gitlab/runner-6fcciz8x-project-48-concurrent-0l5cjf to be running, status is Pending
Waiting for pod gitlab/runner-6fcciz8x-project-48-concurrent-0l5cjf to be running, status is Pending
Waiting for pod gitlab/runner-6fcciz8x-project-48-concurrent-0l5cjf to be running, status is Pending
Waiting for pod gitlab/runner-6fcciz8x-project-48-concurrent-0l5cjf to be running, status is Pending
Waiting for pod gitlab/runner-6fcciz8x-project-48-concurrent-0l5cjf to be running, status is Pending
Waiting for pod gitlab/runner-6fcciz8x-project-48-concurrent-0l5cjf to be running, status is Pending
ERROR: Job failed: image pull failed: Back-off pulling image "registry.***.com/gitlab-ci/golang-ci:latest"
ERROR: Job failed: image pull failed: rpc error: code = Unknown desc = Error response from daemon: Get https://registry.***.com/v2/: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)
ERROR: Job failed: image pull failed: [rpc error: code = Unknown desc = Error response from daemon: Get https://registry.***.com/v2/: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers), rpc error: code = Unknown desc = Error response from daemon: Get https://registry.***.com/v2/: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)]
Edited by 🤖 GitLab Bot 🤖