gitlab-runner api/v4/runners: x509: certificate signed by unknown authority
Summary
Hey there! I'm trying to install gitlab-ce in my GKE with helm chart, but got gitlab-runner stuck with certificate signed by unknown authority error. Other pods and services are working just fine. GKE created by terraform (pretty much default config from default terraform example). Also I created Cloud DNS zone with information for fiction azakaka.com domain to prevent issues with connectivity. (I tried with real domain, the problem is the same) I don't have own wildcard or any other certificate, so I'm using built-in cert-manager.
Steps to reproduce
helm upgrade --install gitlab gitlab/gitlab \
--set global.hosts.domain=azakaka.com \
--set certmanager-issuer.email=me@azakaka.com \
--set rbac.create=true \
--set gitlabUrl=https://gitlab.azakaka.com,runnerRegistrationToken=sjPyymRWNH8L5YXz-YiG \
--set global.hosts.externalIP=104.198.210.133 \
--set global.edition=ce Actual behavior
ERROR: Registering runner... failed runner=9Hcilm8k status=couldn't execute POST against https://gitlab.azakaka.com/api/v4/runners: Post https://gitlab.azakaka.com/api/v4/runners: x509: certificate signed by unknown authority
Expected behavior
gitlab-runner accepts an automaticly generated certificate and connects to gitlab
Relevant logs and/or screenshots
kubectl describe pods:
Name: gitlab-gitlab-runner-75c6dcd8-9rqmx
Namespace: default
Priority: 0
Node: gke-cluster-0-default-pool-57adabee-jm3x/10.128.15.231
Start Time: Thu, 09 Apr 2020 21:19:50 -0700
Labels: app=gitlab-gitlab-runner
chart=gitlab-runner-0.15.0
heritage=Tiller
pod-template-hash=75c6dcd8
release=gitlab
Annotations: checksum/configmap: 9a9e828267d914631bc4ad06b1ae17eafed3f6b339a5530a525c84f85bb10502
checksum/secrets: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
kubernetes.io/limit-ranger: LimitRanger plugin set: cpu request for container gitlab-gitlab-runner; cpu request for init container configure
prometheus.io/port: 9252
prometheus.io/scrape: true
Status: Running
IP: 10.12.0.12
Controlled By: ReplicaSet/gitlab-gitlab-runner-75c6dcd8
Init Containers:
configure:
Container ID: docker://8d9fd01cab34e278991b1c3df11da4f83ff8b5a8aba872a41caef844f2af854f
Image: gitlab/gitlab-runner:alpine-v12.9.0
Image ID: docker-pullable://gitlab/gitlab-runner@sha256:d9b6b40e696cf4f4f443f97920f399cef5664cdf14ec18b8761ba5ac36d7b092
Port: <none>
Host Port: <none>
Command:
sh
/config/configure
State: Terminated
Reason: Completed
Exit Code: 0
Started: Thu, 09 Apr 2020 21:20:20 -0700
Finished: Thu, 09 Apr 2020 21:20:20 -0700
Ready: True
Restart Count: 0
Requests:
cpu: 100m
Environment:
CI_SERVER_URL: https://gitlab.azakaka.com
CLONE_URL:
RUNNER_REQUEST_CONCURRENCY: 1
RUNNER_EXECUTOR: kubernetes
REGISTER_LOCKED: false
RUNNER_TAG_LIST:
RUNNER_OUTPUT_LIMIT: 4096
KUBERNETES_IMAGE: ubuntu:16.04
KUBERNETES_NAMESPACE: default
KUBERNETES_POLL_TIMEOUT: 180
KUBERNETES_CPU_LIMIT:
KUBERNETES_MEMORY_LIMIT:
KUBERNETES_CPU_REQUEST:
KUBERNETES_MEMORY_REQUEST:
KUBERNETES_SERVICE_ACCOUNT:
KUBERNETES_SERVICE_CPU_LIMIT:
KUBERNETES_SERVICE_MEMORY_LIMIT:
KUBERNETES_SERVICE_CPU_REQUEST:
KUBERNETES_SERVICE_MEMORY_REQUEST:
KUBERNETES_HELPER_CPU_LIMIT:
KUBERNETES_HELPER_MEMORY_LIMIT:
KUBERNETES_HELPER_CPU_REQUEST:
KUBERNETES_HELPER_MEMORY_REQUEST:
KUBERNETES_HELPER_IMAGE:
KUBERNETES_PULL_POLICY:
CACHE_TYPE: s3
CACHE_PATH: gitlab-runner
CACHE_SHARED: true
CACHE_S3_SERVER_ADDRESS: minio.azakaka.com
CACHE_S3_BUCKET_NAME: runner-cache
CACHE_S3_BUCKET_LOCATION: us-east-1
Mounts:
/config from scripts (ro)
/init-secrets from init-runner-secrets (ro)
/secrets from runner-secrets (rw)
/var/run/secrets/kubernetes.io/serviceaccount from gitlab-gitlab-runner-token-lnlh2 (ro)
Containers:
gitlab-gitlab-runner:
Container ID: docker://a7d284bf197646221d644fb142cd79ffe23d5c1544fdc560b634a36534771a00
Image: gitlab/gitlab-runner:alpine-v12.9.0
Image ID: docker-pullable://gitlab/gitlab-runner@sha256:d9b6b40e696cf4f4f443f97920f399cef5664cdf14ec18b8761ba5ac36d7b092
Port: 9252/TCP
Host Port: 0/TCP
Command:
/bin/bash
/scripts/entrypoint
State: Running
Started: Thu, 09 Apr 2020 21:22:53 -0700
Last State: Terminated
Reason: Error
Exit Code: 1
Started: Thu, 09 Apr 2020 21:20:22 -0700
Finished: Thu, 09 Apr 2020 21:22:52 -0700
Ready: False
Restart Count: 1
Requests:
cpu: 100m
Liveness: exec [/bin/bash /scripts/check-live] delay=60s timeout=1s period=10s #success=1 #failure=3
Readiness: exec [/usr/bin/pgrep gitlab.*runner] delay=10s timeout=1s period=10s #success=1 #failure=3
Environment:
CI_SERVER_URL: https://gitlab.azakaka.com
CLONE_URL:
RUNNER_REQUEST_CONCURRENCY: 1
RUNNER_EXECUTOR: kubernetes
REGISTER_LOCKED: false
RUNNER_TAG_LIST:
RUNNER_OUTPUT_LIMIT: 4096
KUBERNETES_IMAGE: ubuntu:16.04
KUBERNETES_NAMESPACE: default
KUBERNETES_POLL_TIMEOUT: 180
KUBERNETES_CPU_LIMIT:
KUBERNETES_MEMORY_LIMIT:
KUBERNETES_CPU_REQUEST:
KUBERNETES_MEMORY_REQUEST:
KUBERNETES_SERVICE_ACCOUNT:
KUBERNETES_SERVICE_CPU_LIMIT:
KUBERNETES_SERVICE_MEMORY_LIMIT:
KUBERNETES_SERVICE_CPU_REQUEST:
KUBERNETES_SERVICE_MEMORY_REQUEST:
KUBERNETES_HELPER_CPU_LIMIT:
KUBERNETES_HELPER_MEMORY_LIMIT:
KUBERNETES_HELPER_CPU_REQUEST:
KUBERNETES_HELPER_MEMORY_REQUEST:
KUBERNETES_HELPER_IMAGE:
KUBERNETES_PULL_POLICY:
CACHE_TYPE: s3
CACHE_PATH: gitlab-runner
CACHE_SHARED: true
CACHE_S3_SERVER_ADDRESS: minio.azakaka.com
CACHE_S3_BUCKET_NAME: runner-cache
CACHE_S3_BUCKET_LOCATION: us-east-1
Mounts:
/home/gitlab-runner/.gitlab-runner from etc-gitlab-runner (rw)
/scripts from scripts (rw)
/secrets from runner-secrets (rw)
/var/run/secrets/kubernetes.io/serviceaccount from gitlab-gitlab-runner-token-lnlh2 (ro)
Conditions:
Type Status
Initialized True
Ready False
ContainersReady False
PodScheduled True
Volumes:
runner-secrets:
Type: EmptyDir (a temporary directory that shares a pod's lifetime)
Medium: Memory
SizeLimit: <unset>
etc-gitlab-runner:
Type: EmptyDir (a temporary directory that shares a pod's lifetime)
Medium: Memory
SizeLimit: <unset>
init-runner-secrets:
Type: Projected (a volume that contains injected data from multiple sources)
SecretName: gitlab-minio-secret
SecretOptionalName: <nil>
SecretName: gitlab-gitlab-runner-secret
SecretOptionalName: <nil>
scripts:
Type: ConfigMap (a volume populated by a ConfigMap)
Name: gitlab-gitlab-runner
Optional: false
gitlab-gitlab-runner-token-lnlh2:
Type: Secret (a volume populated by a Secret)
SecretName: gitlab-gitlab-runner-token-lnlh2
Optional: false
QoS Class: Burstable
Node-Selectors: <none>
Tolerations: node.kubernetes.io/not-ready:NoExecute for 300s
node.kubernetes.io/unreachable:NoExecute for 300s
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Scheduled 4m28s default-scheduler Successfully assigned default/gitlab-gitlab-runner-75c6dcd8-9rqmx to gke-cluster-0-default-pool-57adabee-jm3x
Warning FailedMount 4m27s kubelet, gke-cluster-0-default-pool-57adabee-jm3x MountVolume.SetUp failed for volume "gitlab-gitlab-runner-token-lnlh2" : couldn't propagate object cache: timed out waiting for the condition
Warning FailedMount 4m27s kubelet, gke-cluster-0-default-pool-57adabee-jm3x MountVolume.SetUp failed for volume "scripts" : couldn't propagate object cache: timed out waiting for the condition
Normal Pulling 4m24s kubelet, gke-cluster-0-default-pool-57adabee-jm3x Pulling image "gitlab/gitlab-runner:alpine-v12.9.0"
Normal Pulled 3m59s kubelet, gke-cluster-0-default-pool-57adabee-jm3x Successfully pulled image "gitlab/gitlab-runner:alpine-v12.9.0"
Normal Created 3m59s kubelet, gke-cluster-0-default-pool-57adabee-jm3x Created container configure
Normal Started 3m58s kubelet, gke-cluster-0-default-pool-57adabee-jm3x Started container configure
Warning Unhealthy 90s (x13 over 3m40s) kubelet, gke-cluster-0-default-pool-57adabee-jm3x Readiness probe failed:
Normal Pulled 86s (x2 over 3m57s) kubelet, gke-cluster-0-default-pool-57adabee-jm3x Container image "gitlab/gitlab-runner:alpine-v12.9.0" already present on machine
Normal Created 86s (x2 over 3m56s) kubelet, gke-cluster-0-default-pool-57adabee-jm3x Created container gitlab-gitlab-runner
Normal Started 85s (x2 over 3m56s) kubelet, gke-cluster-0-default-pool-57adabee-jm3x Started container gitlab-gitlab-runnerkubectl logs:
ERROR: Registering runner... failed runner=9Hcilm8k status=couldn't execute POST against https://gitlab.azakaka.com/api/v4/runners: Post https://gitlab.azakaka.com/api/v4/runners: x509: certificate signed by unknown authority
PANIC: Failed to register this runner. Perhaps you are having network problems
Registration attempt 29 of 30
Runtime platform arch=amd64 os=linux pid=404 revision=4c96e5ad version=12.9.0
WARNING: Running in user-mode.
WARNING: The user-mode requires you to manually start builds processing:
WARNING: $ gitlab-runner run
WARNING: Use sudo for system-mode:
WARNING: $ sudo gitlab-runner...
ERROR: Registering runner... failed runner=9Hcilm8k status=couldn't execute POST against https://gitlab.azakaka.com/api/v4/runners: Post https://gitlab.azakaka.com/api/v4/runners: x509: certificate signed by unknown authority
PANIC: Failed to register this runner. Perhaps you are having network problems
Registration attempt 30 of 30
Runtime platform arch=amd64 os=linux pid=424 revision=4c96e5ad version=12.9.0
WARNING: Running in user-mode.
WARNING: The user-mode requires you to manually start builds processing:
WARNING: $ gitlab-runner run
WARNING: Use sudo for system-mode:
WARNING: $ sudo gitlab-runner...
ERROR: Registering runner... failed runner=9Hcilm8k status=couldn't execute POST against https://gitlab.azakaka.com/api/v4/runners: Post https://gitlab.azakaka.com/api/v4/runners: x509: certificate signed by unknown authority
PANIC: Failed to register this runner. Perhaps you are having network problems ``Secrets
kubectl get secrets
NAME TYPE DATA AGE
default-token-4p89f kubernetes.io/service-account-token 3 36m
gitlab-acme-key Opaque 1 29m
gitlab-cainjector-token-rcbqm kubernetes.io/service-account-token 3 29m
gitlab-cert-manager-token-q89sc kubernetes.io/service-account-token 3 29m
gitlab-cert-manager-webhook-ca kubernetes.io/tls 3 29m
gitlab-cert-manager-webhook-tls kubernetes.io/tls 3 29m
gitlab-certmanager-issuer-token-z2w2q kubernetes.io/service-account-token 3 29m
gitlab-gitaly-secret Opaque 1 30m
gitlab-gitlab-initial-root-password Opaque 1 30m
gitlab-gitlab-runner-secret Opaque 2 30m
gitlab-gitlab-runner-token-lnlh2 kubernetes.io/service-account-token 3 29m
gitlab-gitlab-shell-host-keys Opaque 8 30m
gitlab-gitlab-shell-secret Opaque 1 30m
gitlab-gitlab-tls kubernetes.io/tls 3 28m
gitlab-gitlab-workhorse-secret Opaque 1 30m
gitlab-minio-secret Opaque 2 30m
gitlab-minio-tls kubernetes.io/tls 3 28m
gitlab-nginx-ingress-token-rfvcb kubernetes.io/service-account-token 3 29m
gitlab-postgresql-password Opaque 2 30m
gitlab-prometheus-server-token-96xkj kubernetes.io/service-account-token 3 29m
gitlab-rails-secret Opaque 1 30m
gitlab-redis-secret Opaque 1 30m
gitlab-registry-httpsecret Opaque 1 30m
gitlab-registry-secret Opaque 2 30m
gitlab-registry-tls kubernetes.io/tls 3 28mEnvironment description
The enviroment is GKE and helm + newer gitlab chart kubectl version
Client Version: version.Info{Major:"1", Minor:"15", GitVersion:"v1.15.3", GitCommit:"2d3c76f9091b6bec110a5e63777c332469e0cba2", GitTreeState:"clean", BuildDate:"2019-08-19T11:13:54Z", GoVersion:"go1.12.9", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"14+", GitVersion:"v1.14.10-gke.27", GitCommit:"145f9e21a4515947d6fb10819e5a336aff1b6959", GitTreeState:"clean", BuildDate:"2020-02-21T18:01:40Z", GoVersion:"go1.12.12b4", Compiler:"gc", Platform:"linux/amd64"}helm version
Client: &version.Version{SemVer:"v2.16.5", GitCommit:"89bd14c1541fa93a09492010030fd3699ca65a97", GitTreeState:"clean"}
Server: &version.Version{SemVer:"v2.16.5", GitCommit:"89bd14c1541fa93a09492010030fd3699ca65a97", GitTreeState:"clean"}Used GitLab Runner version
gitlab-runner --version
Version: 12.9.0
Git revision: 4c96e5ad
Git branch: 12-9-stable
GO version: go1.13.8
Built: 2020-03-20T13:01:56+0000
OS/Arch: linux/amd64Possible fixes
I found arcticle (https://docs.gitlab.com/runner/configuration/tls-self-signed.html#supported-options-for-self-signed-certificates) regarding this issue, but it's unclear for me how to fix this issue. I've got no srt file. Where should I get one? At gitlab-runner' pod I checked folders /etc/gitlab-runner/certs/ and ~/.gitlab-runner/certs/ - both empty.
Is there any way to request certificate from cert manager manualy to copy it to required folders?