Skip to content

Kubernetes executor fails to pull image from internal docker registry when job is based on it

Summary

After exactly 1 month of usage gitlab-runner started failing with ERROR: Job failed: image pull failed: Back-off pulling image "registry.***.com/gitlab-ci/golang-ci:latest" when the image present.

Steps to reproduce

Install GitLab in a Kubernetes Cluster with Helm and use for a month :)

Configuration used

values.yaml

global:
  hosts:
    domain: ***
    externalIP: ***

  appConfig:
    omniauth:
      enabled: true
      blockAutoCreatedUsers: false
      autoLinkSamlUser: true
      allowSingleSignOn: ['saml']
      syncProfileFromProvider: ['saml']
      providers:
        - secret: ssoprovider
    defaultProjectsFeatures:
      issues: false
      wiki: false
      snippets: false

gitlab-runner:
  runners:
    privileged: true
  envVars:
    - name: RUNNER_BUILDS_DIR
      value: /builds

.gitlab-ci.yml example snippet

.golang-lint:
  image: registry.***.com/gitlab-ci/golang-ci:latest
  script:
    - go vet ./...

Current behavior

Kubernetes Executor fails to run jobs based on internal images.

Expected behavior

Kubernetes Executor not failing to run jobs based on internal images.

Versions

  • Chart: v1.6.3
  • Platform:
    • Cloud: GKE
  • Kubernetes:
    • Client: version.Info{Major:"1", Minor:"13", GitVersion:"v1.13.4", GitCommit:"c27b913fddd1a6c480c229191a087698aa92f0b1", GitTreeState:"clean", BuildDate:"2019-02-28T13:37:52Z", GoVersion:"go1.11.5", Compiler:"gc", Platform:"linux/amd64"}
    • Server: version.Info{Major:"1", Minor:"11+", GitVersion:"v1.11.7-gke.12", GitCommit:"06f08e60069231bd21bdf673cf0595aac80b99f6", GitTreeState:"clean", BuildDate:"2019-02-25T20:37:10Z", GoVersion:"go1.10.8b4", Compiler:"gc", Platform:"linux/amd64"}
  • Helm:
    • Client: &version.Version{SemVer:"v2.13.1", GitCommit:"618447cbf203d147601b4b9bd7f8c37a5d39fbb4", GitTreeState:"clean"}
    • Server: &version.Version{SemVer:"v2.13.0", GitCommit:"79d07943b03aea2b76c12644b4b54733bc5958d6", GitTreeState:"clean"}

Relevant logs

Jobs fail in several ways:

Running with gitlab-runner 11.8.0 (4745a6f3)
  on gitlab-gitlab-runner-59b4dd5899-tzk7g 6Fcciz8x
Using Kubernetes namespace: gitlab
Using Kubernetes executor with image registry.***.com/gitlab-ci/golang-ci:latest ...
Waiting for pod gitlab/runner-6fcciz8x-project-48-concurrent-0l5cjf to be running, status is Pending
Waiting for pod gitlab/runner-6fcciz8x-project-48-concurrent-0l5cjf to be running, status is Pending
Waiting for pod gitlab/runner-6fcciz8x-project-48-concurrent-0l5cjf to be running, status is Pending
Waiting for pod gitlab/runner-6fcciz8x-project-48-concurrent-0l5cjf to be running, status is Pending
Waiting for pod gitlab/runner-6fcciz8x-project-48-concurrent-0l5cjf to be running, status is Pending
Waiting for pod gitlab/runner-6fcciz8x-project-48-concurrent-0l5cjf to be running, status is Pending
ERROR: Job failed: image pull failed: Back-off pulling image "registry.***.com/gitlab-ci/golang-ci:latest"
ERROR: Job failed: image pull failed: rpc error: code = Unknown desc = Error response from daemon: Get https://registry.***.com/v2/: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)
ERROR: Job failed: image pull failed: [rpc error: code = Unknown desc = Error response from daemon: Get https://registry.***.com/v2/: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers), rpc error: code = Unknown desc = Error response from daemon: Get https://registry.***.com/v2/: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)]
Edited by 🤖 GitLab Bot 🤖