Docker services docker-dind does not start - Exit Code 255 (microK8s)
Hello everyone and thanks in advance,
We have a pipeline to build a docker, but the docker service doesn't start and the job fails.
build-docker:
stage: build
image: docker:19.03.0-dind
<<: *default_rules
variables:
DOCKER_DRIVER: overlay2
DOCKER_TLS_CERTDIR: ''
DOCKER_HOST: tcp://docker:2375
services:
- name: docker:19.03.0-dind
alias: docker
command: [ "--tls=false" ]
before_script:
- until docker info; do sleep 1; done
script:
- until docker info; do sleep 1; done #
- export CI_APPLICATION_TAG=$VERSION
- docker login -u $CI_REGISTRY_USER -p $CI_REGISTRY_PASSWORD $CI_REGISTRY
- docker build -t $CI_APPLICATION_REPOSITORY:$CI_APPLICATION_TAG .
- docker push $CI_APPLICATION_REPOSITORY:$CI_APPLICATION_TAG
needs:
- prepareI read various issues but to no avail, it seems to me that it is a problem related to the creation of the pod. gitlab-org/charts/gitlab-runner#352
Several times I get this error MountVolume.SetUp failed for volume "projected-secrets" : secret "gitlab-runner" not found and looking in the issues I could not find a solution.
When job started:
After few seconds:
microk8s kubectl describe pod runner-fgsdjy6q-project-194-concurrent-0mkv6v -n gitlab-managed-apps
Name: runner-fgsdjy6q-project-194-concurrent-0mkv6v
Namespace: gitlab-managed-apps
Priority: 0
Service Account: default
Node: microk8s/10.60.10.155
Start Time: Thu, 29 Jun 2023 07:48:34 +0000
Labels: pod=runner-fgsdjy6q-project-194-concurrent-0
Annotations: cni.projectcalico.org/containerID: 4d9adc2bafef490872d42ad21fdc8037e72dded5609c9cfd9944ba4413840a95
cni.projectcalico.org/podIP: 10.1.128.205/32
cni.projectcalico.org/podIPs: 10.1.128.205/32
job.runner.gitlab.com/before_sha: 6fd9cd70cba8f2b096109c247c9d198de812128c
job.runner.gitlab.com/id: 16187
job.runner.gitlab.com/name: build-docker
job.runner.gitlab.com/ref: pre-release
job.runner.gitlab.com/sha: 28022be11e729cc6eadf6050e25817c3adf7d76a
job.runner.gitlab.com/url: #Ometted#
project.runner.gitlab.com/id: 194
Status: Running
IP: 10.1.128.205
IPs:
IP: 10.1.128.205
Init Containers:
init-permissions:
Container ID: containerd://9adc8c0159465712e7909bf63fc2d64a92f3dfdb778e0d384d36c38acb63744f
Image: registry.gitlab.com/gitlab-org/gitlab-runner/gitlab-runner-helper:x86_64-b72e108d
Image ID: registry.gitlab.com/gitlab-org/gitlab-runner/gitlab-runner-helper@sha256:44d67e90cdab34d37992ea1b87273817d30aa28fe5783e110eeab2d9f0a313a7
Port: <none>
Host Port: <none>
Command:
sh
-c
touch /logs-194-16187/output.log && (chmod 777 /logs-194-16187/output.log || exit 0)
State: Terminated
Reason: Completed
Exit Code: 0
Started: Thu, 29 Jun 2023 07:48:36 +0000
Finished: Thu, 29 Jun 2023 07:48:36 +0000
Ready: True
Restart Count: 0
Environment: <none>
Mounts:
/builds from repo (rw)
/logs-194-16187 from logs (rw)
/scripts-194-16187 from scripts (rw)
/var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-prxk9 (ro)
Containers:
build:
Container ID: containerd://50e9ca64366710c2a7cd59c8b36b2f4a4455b326a2099b58600feb281fb0ec9e
Image: docker:19.03.0-dind
Image ID: docker.io/library/docker@sha256:442ac4b31375cbe617f31759b5199d240f11d5f430e54946575b274b2fb6f096
Port: <none>
Host Port: <none>
Command:
sh
-c
if [ -x /usr/local/bin/bash ]; then
exec /usr/local/bin/bash
elif [ -x /usr/bin/bash ]; then
exec /usr/bin/bash
elif [ -x /bin/bash ]; then
exec /bin/bash
elif [ -x /usr/local/bin/sh ]; then
exec /usr/local/bin/sh
elif [ -x /usr/bin/sh ]; then
exec /usr/bin/sh
elif [ -x /bin/sh ]; then
exec /bin/sh
elif [ -x /busybox/sh ]; then
exec /busybox/sh
else
echo shell not found
exit 1
fi
State: Running
Started: Thu, 29 Jun 2023 07:48:37 +0000
Ready: True
Restart Count: 0
Environment:
FF_CMD_DISABLE_DELAYED_ERROR_LEVEL_EXPANSION: false
FF_NETWORK_PER_BUILD: false
FF_USE_LEGACY_KUBERNETES_EXECUTION_STRATEGY: false
FF_USE_DIRECT_DOWNLOAD: true
FF_SKIP_NOOP_BUILD_STAGES: true
FF_USE_FASTZIP: false
FF_DISABLE_UMASK_FOR_DOCKER_EXECUTOR: false
FF_ENABLE_BASH_EXIT_CODE_CHECK: false
FF_USE_WINDOWS_LEGACY_PROCESS_STRATEGY: true
FF_USE_NEW_BASH_EVAL_STRATEGY: false
FF_USE_POWERSHELL_PATH_RESOLVER: false
FF_USE_DYNAMIC_TRACE_FORCE_SEND_INTERVAL: false
FF_SCRIPT_SECTIONS: false
FF_USE_NEW_SHELL_ESCAPE: false
FF_ENABLE_JOB_CLEANUP: false
FF_KUBERNETES_HONOR_ENTRYPOINT: false
FF_POSIXLY_CORRECT_ESCAPES: false
FF_USE_IMPROVED_URL_MASKING: false
FF_RESOLVE_FULL_TLS_CHAIN: true
FF_DISABLE_POWERSHELL_STDIN: false
FF_USE_POD_ACTIVE_DEADLINE_SECONDS: false
FF_USE_ADVANCED_POD_SPEC_CONFIGURATION: false
FF_SET_PERMISSIONS_BEFORE_CLEANUP: true
FF_SECRET_RESOLVING_FAILS_IF_MISSING: true
CI_JOB_IMAGE: docker:19.03.0-dind
CI_RUNNER_SHORT_TOKEN: FGSdJy6q
CI_BUILDS_DIR: /builds
CI_PROJECT_DIR: #Ometted#
CI_CONCURRENT_ID: 0
CI_CONCURRENT_PROJECT_ID: 0
CI_SERVER: yes
CI_JOB_STATUS: running
CI_JOB_TIMEOUT: 3600
CI_SERVER_TLS_CA_FILE: #Ometted#
CI_PIPELINE_ID: 8565
CI_PIPELINE_URL: #Ometted#
CI_JOB_ID: 16187
CI_JOB_URL: #Ometted#
CI_JOB_STARTED_AT: 2023-06-29T07:48:32Z
CI_REGISTRY_USER: gitlab-ci-token
CI_DEPENDENCY_PROXY_USER: gitlab-ci-token
CI_JOB_NAME: build-docker
CI_JOB_NAME_SLUG: build-docker
CI_JOB_STAGE: build
CI_NODE_TOTAL: 1
CI: true
GITLAB_CI: true
CI_SERVER_URL: #Ometted#
CI_SERVER_HOST: #Ometted#
CI_SERVER_PORT: 443
CI_SERVER_PROTOCOL: https
CI_SERVER_SHELL_SSH_HOST: #Ometted#
CI_SERVER_SHELL_SSH_PORT: 2223
CI_SERVER_NAME: GitLab
CI_SERVER_VERSION: 16.0.5
CI_SERVER_VERSION_MAJOR: 16
CI_SERVER_VERSION_MINOR: 0
CI_SERVER_VERSION_PATCH: 5
CI_SERVER_REVISION: 6e840c5468f
GITLAB_FEATURES:
CI_PROJECT_ID: 194
CI_PROJECT_NAME: #Ometted#
CI_PROJECT_TITLE: #Ometted#
CI_PROJECT_DESCRIPTION:
CI_PROJECT_PATH: #Ometted#
CI_PROJECT_PATH_SLUG: #Ometted#
CI_PROJECT_NAMESPACE: #Ometted#
CI_PROJECT_NAMESPACE_ID: 354
CI_PROJECT_ROOT_NAMESPACE: #Ometted#
CI_PROJECT_URL: #Ometted#
CI_PROJECT_VISIBILITY: private
CI_PROJECT_REPOSITORY_LANGUAGES:
CI_PROJECT_CLASSIFICATION_LABEL:
CI_DEFAULT_BRANCH: main
CI_CONFIG_PATH: .gitlab-ci.yml
CI_PAGES_DOMAIN: #Ometted#
CI_PAGES_URL: #Ometted#
CI_REGISTRY: #Ometted#
CI_REGISTRY_IMAGE: #Ometted#
CI_DEPENDENCY_PROXY_SERVER: #Ometted#
CI_DEPENDENCY_PROXY_GROUP_IMAGE_PREFIX: #Ometted#
CI_DEPENDENCY_PROXY_DIRECT_GROUP_IMAGE_PREFIX: #Ometted#
CI_API_V4_URL: #Ometted#
CI_API_GRAPHQL_URL: #Ometted#
CI_TEMPLATE_REGISTRY_HOST: registry.gitlab.com
CI_PIPELINE_IID: 17
CI_PIPELINE_SOURCE: push
CI_PIPELINE_CREATED_AT: 2023-06-27T19:11:05Z
CI_COMMIT_SHA: 28022be11e729cc6eadf6050e25817c3adf7d76a
CI_COMMIT_SHORT_SHA: 28022be1
CI_COMMIT_BEFORE_SHA: 6fd9cd70cba8f2b096109c247c9d198de812128c
CI_COMMIT_REF_NAME: pre-release
CI_COMMIT_REF_SLUG: pre-release
CI_COMMIT_BRANCH: pre-release
CI_COMMIT_MESSAGE: Wip images
CI_COMMIT_TITLE: Wip images
CI_COMMIT_DESCRIPTION:
CI_COMMIT_REF_PROTECTED: false
CI_COMMIT_TIMESTAMP: 2023-06-27T21:08:34+02:00
CI_COMMIT_AUTHOR: Riccardo Cucia #Ometted#
CI_RUNNER_ID: 2733
CI_RUNNER_DESCRIPTION: runner-gitlab-runner-65c7d564ff-hchrj
CI_RUNNER_TAGS: ["kubernetes", "cluster"]
DOCKER_DRIVER: overlay2
DOCKER_TLS_CERTDIR:
DOCKER_HOST: tcp://docker:2375
ROLLOUT_RESOURCE_TYPE: deployment
CI_REGISTRY: #Ometted#
CI_REGISTRY_IMAGE: #Ometted#
CI_REGISTRY_IMAGE_PRODUCTION: #Ometted#
PRE_RELEASE_BRANCH: pre-release
GITLAB_USER_ID: 2
GITLAB_USER_EMAIL: #Ometted#
GITLAB_USER_LOGIN: #Ometted#
GITLAB_USER_NAME: #Ometted#
CI_DISPOSABLE_ENVIRONMENT: true
CI_RUNNER_VERSION: 16.1.0
CI_RUNNER_REVISION: b72e108d
CI_RUNNER_EXECUTABLE_ARCH: linux/amd64
RUNNER_TEMP_PROJECT_DIR: #Ometted#
Mounts:
/builds from repo (rw)
/logs-194-16187 from logs (rw)
/scripts-194-16187 from scripts (rw)
/var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-prxk9 (ro)
helper:
Container ID: containerd://e880b6301a361d449baa899c6fb28342cc31ed44fb63c90f169f0dd751e3064d
Image: registry.gitlab.com/gitlab-org/gitlab-runner/gitlab-runner-helper:x86_64-b72e108d
Image ID: registry.gitlab.com/gitlab-org/gitlab-runner/gitlab-runner-helper@sha256:44d67e90cdab34d37992ea1b87273817d30aa28fe5783e110eeab2d9f0a313a7
Port: <none>
Host Port: <none>
Command:
sh
-c
if [ -x /usr/local/bin/bash ]; then
exec /usr/local/bin/bash
elif [ -x /usr/bin/bash ]; then
exec /usr/bin/bash
elif [ -x /bin/bash ]; then
exec /bin/bash
elif [ -x /usr/local/bin/sh ]; then
exec /usr/local/bin/sh
elif [ -x /usr/bin/sh ]; then
exec /usr/bin/sh
elif [ -x /bin/sh ]; then
exec /bin/sh
elif [ -x /busybox/sh ]; then
exec /busybox/sh
else
echo shell not found
exit 1
fi
State: Running
Started: Thu, 29 Jun 2023 07:48:37 +0000
Ready: True
Restart Count: 0
Environment: <none>
Mounts:
/builds from repo (rw)
/logs-194-16187 from logs (rw)
/scripts-194-16187 from scripts (rw)
/var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-prxk9 (ro)
svc-0:
Container ID: containerd://77074ed1b9595457b4b0bd77d411a17fb1e005fe76f6b1ff31ceba1abad01fdf
Image: docker:19.03.0-dind
Image ID: docker.io/library/docker@sha256:442ac4b31375cbe617f31759b5199d240f11d5f430e54946575b274b2fb6f096
Port: <none>
Host Port: <none>
Args:
--tls=false
State: Terminated
Reason: Error
Exit Code: 255
Started: Thu, 29 Jun 2023 07:48:38 +0000
Finished: Thu, 29 Jun 2023 07:48:38 +0000
Ready: False
Restart Count: 0
Environment:
FF_CMD_DISABLE_DELAYED_ERROR_LEVEL_EXPANSION: false
FF_NETWORK_PER_BUILD: false
FF_USE_LEGACY_KUBERNETES_EXECUTION_STRATEGY: false
FF_USE_DIRECT_DOWNLOAD: true
FF_SKIP_NOOP_BUILD_STAGES: true
FF_USE_FASTZIP: false
FF_DISABLE_UMASK_FOR_DOCKER_EXECUTOR: false
FF_ENABLE_BASH_EXIT_CODE_CHECK: false
FF_USE_WINDOWS_LEGACY_PROCESS_STRATEGY: true
FF_USE_NEW_BASH_EVAL_STRATEGY: false
FF_USE_POWERSHELL_PATH_RESOLVER: false
FF_USE_DYNAMIC_TRACE_FORCE_SEND_INTERVAL: false
FF_SCRIPT_SECTIONS: false
FF_USE_NEW_SHELL_ESCAPE: false
FF_ENABLE_JOB_CLEANUP: false
FF_KUBERNETES_HONOR_ENTRYPOINT: false
FF_POSIXLY_CORRECT_ESCAPES: false
FF_USE_IMPROVED_URL_MASKING: false
FF_RESOLVE_FULL_TLS_CHAIN: true
FF_DISABLE_POWERSHELL_STDIN: false
FF_USE_POD_ACTIVE_DEADLINE_SECONDS: false
FF_USE_ADVANCED_POD_SPEC_CONFIGURATION: false
FF_SET_PERMISSIONS_BEFORE_CLEANUP: true
FF_SECRET_RESOLVING_FAILS_IF_MISSING: true
CI_JOB_IMAGE: docker:19.03.0-dind
CI_RUNNER_SHORT_TOKEN: FGSdJy6q
CI_BUILDS_DIR: /builds
CI_PROJECT_DIR: #Ometted#
CI_CONCURRENT_ID: 0
CI_CONCURRENT_PROJECT_ID: 0
CI_SERVER: yes
CI_JOB_STATUS: running
CI_JOB_TIMEOUT: 3600
CI_SERVER_TLS_CA_FILE: #Ometted#
CI_PIPELINE_ID: 8565
CI_PIPELINE_URL: #Ometted#
CI_JOB_ID: 16187
CI_JOB_URL: #Ometted#
CI_JOB_STARTED_AT: 2023-06-29T07:48:32Z
CI_REGISTRY_USER: gitlab-ci-token
CI_DEPENDENCY_PROXY_USER: gitlab-ci-token
CI_JOB_NAME: build-docker
CI_JOB_NAME_SLUG: build-docker
CI_JOB_STAGE: build
CI_NODE_TOTAL: 1
CI: true
GITLAB_CI: true
CI_SERVER_URL: #Ometted#
CI_SERVER_HOST: #Ometted#
CI_SERVER_PORT: 443
CI_SERVER_PROTOCOL: https
CI_SERVER_SHELL_SSH_HOST: #Ometted#
CI_SERVER_SHELL_SSH_PORT: 2223
CI_SERVER_NAME: GitLab
CI_SERVER_VERSION: 16.0.5
CI_SERVER_VERSION_MAJOR: 16
CI_SERVER_VERSION_MINOR: 0
CI_SERVER_VERSION_PATCH: 5
CI_SERVER_REVISION: 6e840c5468f
GITLAB_FEATURES:
CI_PROJECT_ID: 194
CI_PROJECT_NAME: #Ometted#
CI_PROJECT_TITLE: #Ometted#
CI_PROJECT_DESCRIPTION:
CI_PROJECT_PATH: #Ometted#
CI_PROJECT_PATH_SLUG: #Ometted#
CI_PROJECT_NAMESPACE: #Ometted#
CI_PROJECT_NAMESPACE_ID: 354
CI_PROJECT_ROOT_NAMESPACE: #Ometted#
CI_PROJECT_URL: #Ometted#
CI_PROJECT_VISIBILITY: private
CI_PROJECT_REPOSITORY_LANGUAGES:
CI_PROJECT_CLASSIFICATION_LABEL:
CI_DEFAULT_BRANCH: main
CI_CONFIG_PATH: .gitlab-ci.yml
CI_PAGES_DOMAIN: #Ometted#
CI_PAGES_URL: #Ometted#
CI_REGISTRY: #Ometted#
CI_REGISTRY_IMAGE: #Ometted#
CI_DEPENDENCY_PROXY_SERVER: #Ometted#
CI_DEPENDENCY_PROXY_GROUP_IMAGE_PREFIX: #Ometted#
CI_DEPENDENCY_PROXY_DIRECT_GROUP_IMAGE_PREFIX: #Ometted#
CI_API_V4_URL: #Ometted#
CI_API_GRAPHQL_URL: #Ometted#
CI_TEMPLATE_REGISTRY_HOST: registry.gitlab.com
CI_PIPELINE_IID: 17
CI_PIPELINE_SOURCE: push
CI_PIPELINE_CREATED_AT: 2023-06-27T19:11:05Z
CI_COMMIT_SHA: 28022be11e729cc6eadf6050e25817c3adf7d76a
CI_COMMIT_SHORT_SHA: 28022be1
CI_COMMIT_BEFORE_SHA: 6fd9cd70cba8f2b096109c247c9d198de812128c
CI_COMMIT_REF_NAME: pre-release
CI_COMMIT_REF_SLUG: pre-release
CI_COMMIT_BRANCH: pre-release
CI_COMMIT_MESSAGE: Wip images
CI_COMMIT_TITLE: Wip images
CI_COMMIT_DESCRIPTION:
CI_COMMIT_REF_PROTECTED: false
CI_COMMIT_TIMESTAMP: 2023-06-27T21:08:34+02:00
CI_COMMIT_AUTHOR: Riccardo Cucia #Ometted#
CI_RUNNER_ID: 2733
CI_RUNNER_DESCRIPTION: runner-gitlab-runner-65c7d564ff-hchrj
CI_RUNNER_TAGS: ["kubernetes", "cluster"]
DOCKER_DRIVER: overlay2
DOCKER_TLS_CERTDIR:
DOCKER_HOST: tcp://docker:2375
ROLLOUT_RESOURCE_TYPE: deployment
CI_REGISTRY: #Ometted#
CI_REGISTRY_IMAGE: #Ometted#
CI_REGISTRY_IMAGE_PRODUCTION: #Ometted#
PRE_RELEASE_BRANCH: pre-release
GITLAB_USER_ID: 2
GITLAB_USER_EMAIL: #Ometted#
GITLAB_USER_LOGIN: #Ometted#
GITLAB_USER_NAME: Riccardo
CI_DISPOSABLE_ENVIRONMENT: true
CI_RUNNER_VERSION: 16.1.0
CI_RUNNER_REVISION: b72e108d
CI_RUNNER_EXECUTABLE_ARCH: linux/amd64
RUNNER_TEMP_PROJECT_DIR: /#Ometted#
Mounts:
/builds from repo (rw)
/logs-194-16187 from logs (rw)
/scripts-194-16187 from scripts (rw)
/var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-prxk9 (ro)
Conditions:
Type Status
Initialized True
Ready False
ContainersReady False
PodScheduled True
Volumes:
repo:
Type: EmptyDir (a temporary directory that shares a pod's lifetime)
Medium:
SizeLimit: <unset>
scripts:
Type: EmptyDir (a temporary directory that shares a pod's lifetime)
Medium:
SizeLimit: <unset>
logs:
Type: EmptyDir (a temporary directory that shares a pod's lifetime)
Medium:
SizeLimit: <unset>
kube-api-access-prxk9:
Type: Projected (a volume that contains injected data from multiple sources)
TokenExpirationSeconds: 3607
ConfigMapName: kube-root-ca.crt
ConfigMapOptional: <nil>
DownwardAPI: true
QoS Class: BestEffort
Node-Selectors: <none>
Tolerations: node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Scheduled 2m default-scheduler Successfully assigned gitlab-managed-apps/runner-fgsdjy6q-project-194-concurrent-0mkv6v to microk8s
Normal Pulled 119s kubelet Container image "registry.gitlab.com/gitlab-org/gitlab-runner/gitlab-runner-helper:x86_64-b72e108d" already present on machine
Normal Created 119s kubelet Created container init-permissions
Normal Started 119s kubelet Started container init-permissions
Normal Pulled 118s kubelet Container image "docker:19.03.0-dind" already present on machine
Normal Created 118s kubelet Created container build
Normal Started 118s kubelet Started container build
Normal Pulled 118s kubelet Container image "registry.gitlab.com/gitlab-org/gitlab-runner/gitlab-runner-helper:x86_64-b72e108d" already present on machine
Normal Created 118s kubelet Created container helper
Normal Started 118s kubelet Started container helper
Normal Pulled 118s kubelet Container image "docker:19.03.0-dind" already present on machine
Normal Created 118s kubelet Created container svc-0
Normal Started 117s kubelet Started container svc-0deploying a kubernetes cluster on GCP I have the same problem.
Edited by Riccardo Cucia