Internally generated wildcard CA not available in toolbox
Summary
With a fresh install of GitLab 16.0.1 from chart version 7.0.1, the internally generated self-signed CA is not loaded into the toolbox certificate store. This results in a failure of the backup-utility, which can't validate the certificate on the internal minio instance.
Steps to reproduce
Standard install of chart 7.0.1, with certmanager disabled -- open the toolbox, and manually try to access the minio instance with 's3cmd ls'. The result is an SSL error on certificate validation.
I tried to manually load the wildcard cert using the global.hosts.certificates, and while I can see it loaded in the configuration, it is not added to the certificate store.
I have gitlab 15.9.3 installed by chart 6.9.3, and the internal wildcard certificate is appropriately loaded and accessible.
Configuration used
certmanager:
install: false
gitlab-runner:
install: false
global:
hosts:
domain: dev.williams.localnet
externalIP: 10.0.0.112
gitlab: {}
hostSuffix: null
https: true
kas: {}
minio:
pages: {}
registry: {}
smartcard: {}
ssh: gitlab.dev.williams.localnet
certificates:
customCAs:
- secret: gitlab-wildcard-tls-ca
ingress:
configureCertmanager: false
nginx-ingress:
config:
hsts: false
Current behavior
backup-utility within the toolbox pod fails with messages like:
Bucket not found: ... (multiple)
... and ...
WARNING: Retrying failed request: /1685278863_2023_05_28_16.0.1-ee_gitlab_backup.tar ([SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1131))
... and finally ...
ERROR: Upload of '/srv/gitlab/tmp/backup_tars/1685278863_2023_05_28_16.0.1-ee_gitlab_backup.tar' failed too many times (Last reason: Upload failed for: /1685278863_2023_05_28_16.0.1-ee_gitlab_backup.tar)
Looking in the /srv/gitlab/tmp/backup_tars, a tarball is created, but I don't know how complete it is since the buckets are not accessible.
Expected behavior
Successful backup with the tarball loaded into the minio object store.
Versions
-
Chart: released version 7.0.1 from the helm repo
-
Platform:
- Self-hosted: self-hosted kubernetes installed using k3s
-
Kubernetes: (
kubectl version
)- Client Version: version.Info{Major:"1", Minor:"21", GitVersion:"v1.21.5+k3s2", GitCommit:"724ef700bab896ff252a75e2be996d5f4ff1b842", GitTreeState:"clean", BuildDate:"2021-10-05T19:59:14Z", GoVersion:"go1.16.8", Compiler:"gc", Platform:"linux/amd64"}
- Server Version: version.Info{Major:"1", Minor:"26", GitVersion:"v1.26.4+k3s1", GitCommit:"8d0255af07e95b841952563253d27b0d10bd72f0", GitTreeState:"clean", BuildDate:"2023-04-20T00:33:18Z", GoVersion:"go1.19.8", Compiler:"gc", Platform:"linux/amd64"}
-
Helm: (
helm version
)- version.BuildInfo{Version:"v3.12.0", GitCommit:"c9f554d75773799f72ceef38c51210f1842a1dea", GitTreeState:"clean", GoVersion:"go1.20.3"}