LFS on object storage checkout error: "x509: certificate signed by unknown authority" on Kubernetes executor
Summary
When using Kubernetes executor with a private or internal project LFS objects on object storage are failing to checkout with error "x509: certificate signed by unknown authority".
tls-ca-file
is set to point at custom root CA. This is verified as working as without it runner fails to even start job. Looking at kubernetes runner log it appears container build
is being passed via CI_SERVER_TLS_CA_FILE
the contents of CA file provided.
Steps to reproduce
- Setup GitLab server with LFS objects using object storage and
proxy_download=false
- GitLab server has custom root CA
- Object storage has same custom root CA
- Setup kubernetes executor. Mount custom root CA and set in
tls-ca-file
. - Create Git private or internal repository with LFS objects
- Add simple CI file
.gitlab-ci.yml
url-verification:
stage: test
script:
- echo 'test'
Actual behavior
LFS objects fail to download
NOTE: For some reason no error is occuring when same project is turned to public visibility level
Expected behavior
LFS objects download
Relevant logs and/or screenshots
job log
Running with gitlab-runner 13.4.1 (e95f89a0)
on Kubernetes-Runner-xxx S5HgehbM
Preparing the "kubernetes" executor
00:00
Using Kubernetes namespace: gitlabrunner-dev
Using Kubernetes executor with image xxxxxxxxxxxxxxxxxx ...
Preparing environment
Waiting for pod gitlabrunner-dev/runner-s5hgehbm-project-1717-concurrent-0nncxm to be running, status is Pending
Waiting for pod gitlabrunner-dev/runner-s5hgehbm-project-1717-concurrent-0nncxm to be running, status is Pending
Waiting for pod gitlabrunner-dev/runner-s5hgehbm-project-1717-concurrent-0nncxm to be running, status is Pending
Waiting for pod gitlabrunner-dev/runner-s5hgehbm-project-1717-concurrent-0nncxm to be running, status is Pending
Running on runner-s5hgehbm-project-1717-concurrent-0nncxm via gitlab-runner-5c97d98999-t9lqr...
Getting source from Git repository
Fetching changes with git depth set to 50...
Initialized empty Git repository in /builds/xxx/yyy/.git/
Created fresh repository.
Checking out 0d9871f6 as master...
LFS: Get https://object.xxx.com/lfs-dev/c8/95/a34909dce385b85cee1a943788044859d685e66c002dbf7b28e10abeef20?X-Amz-Expires=600&X-Amz-Date=20201006T043010Z&X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=svcgitlabstoragedev%2F20201006%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-SignedHeaders=host&X-Amz-Signature=012211eb0ff0e374086e8c2d37556f2d8ca4cc948763e90896f8f5774a100b55: x509: certificate signed by unknown authority
LFS: Get https://object.xxx.com/lfs-dev/51/36/82135bc6ae3cda6ffb02830e5488774564f5ffbece9c25ef81ff4b30990c?X-Amz-Expires=600&X-Amz-Date=20201006T043010Z&X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=svcgitlabstoragedev%2F20201006%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-SignedHeaders=host&X-Amz-Signature=a7a587daca7f74979521a22238a7a310c53aaed6d5406f3d9511a5fbb6dc7d79: x509: certificate signed by unknown authority
LFS: Get https://object.xxx.com/lfs-dev/28/97/e1a6677d6dec0b93ecad7c0e4043da3ce82c1ca9c7a10bf6aa2f5411fcc3?X-Amz-Expires=600&X-Amz-Date=20201006T043011Z&X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=svcgitlabstoragedev%2F20201006%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-SignedHeaders=host&X-Amz-Signature=63d14e7955bc3a305af31abf702fd42edcfa23cccf004cace8f26d042748d306: x509: certificate signed by unknown authority
LFS: Get https://object.xxx.com/lfs-dev/b1/c8/38fec0b1c0381cf51efa3a8764245a1965683ebcc351429d485e1bb2ea9f?X-Amz-Expires=600&X-Amz-Date=20201006T043011Z&X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=svcgitlabstoragedev%2F20201006%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-SignedHeaders=host&X-Amz-Signature=6ef497c23a1fdd7705d9013b5feb75e30f7eb2fce0ed802f5780e01e30003507: x509: certificate signed by unknown authority
...
error: failed to fetch some objects from 'https://gitlab-ci-token:[MASKED]@gitdev.zzz.com/xxx/yyy.git/info/lfs'
Cleaning up file based variables
00:01
ERROR: Job failed: command terminated with exit code 1
Environment description
Kubernetes executor 13.4.1
config.toml contents
concurrent = 20
listen_address = ":9252"
[[ runners ]]
name = "Kubernetes-Runner-xxx"
url = "https://gitdev.zzz.com/ci/"
tls-ca-file = "/mnt/MyRootCAG2.pem"
token = "xxx"
executor = "kubernetes"
[runners.kubernetes]
namespace = "gitlabrunner-dev"
poll_timeout = 600
image_pull_secrets = ["artifactory"]
image = "xxxxxxxxxxxxxxxxxx "
pull_policy = "always"
helper_image = "xxxx/gitlab/gitlab-runner-helper:x86_64-${CI_RUNNER_REVISION}"
[runners.cache]
Type = "s3"
...
[runners.cache.s3]
...
Used GitLab Runner version
Running with gitlab-runner 13.4.1 (e95f89a0)
Using Kubernetes executor with image xxxx ...
Proposal
After #4125 (closed) gets fixed, we should update our documentation at https://docs.gitlab.com/runner/configuration/tls-self-signed.html#kubernetes showing how to use https://docs.gitlab.com/runner/executors/kubernetes.html#using-volumes to set up certificates. Something like:
...
[[runners.kubernetes.volumes.config_map]]
name = "ca-certs"
mount_path = "/etc/gitlab-runner/certs"
read_only = true
[runners.kubernetes.volumes.config_map.items]
"my-ca-cert.pem" = "ca.crt"
...