Troubleshooting custom TLS Certificate Authorities in cloud-native deployments
Summary
see discussion in comments - issue is a placeholder for adding troubleshooting steps to docs
I'm investigating a customer issue (ticket for GitLab team members) relating to TLS.
Based on the issues they're observing, I believe that Rails isn't trusting their corporate root, but Golang is.
They see the classic unable to get local issuer certificate
errors.
Accessing job logs from their private HTTPS object storage server:
- Rails fails with a 500 when accessing the rendered job log
- Downloading the raw job log (with proxy_download set true) works (this is handled by Workhorse)
One potential issue I've noticed:
bundle-certificates calls update-ca-certificates
In a number of places, that script only looks for files matching *.crt
echo "Updating certificates in $ETCCERTSDIR..."
# Add default certificate authorities if requested
if [ "$default" = 1 ]; then
find -L "$CERTSDIR" -type f -name '*.crt' | sort | while read crt
do
add "$crt"
and (upstream)
# Now process certificate authorities installed by the local system
# administrator.
if [ -d "$LOCALCERTSDIR" ]
then
find -L "$LOCALCERTSDIR" -type f -name '*.crt' | sort | while read crt
do
add "$crt"
Result: a certificate uploaded as a .pem
file would not be processed.
# sha256sum /usr/local/share/ca-certificates/myroot.pem
58fd68402402917b85858cf61952593a17c8b559991ab969ae9e4dd016472b83 /usr/local/share/ca-certificates/myroot.pem
# update-ca-certificates --fresh
[snip]
# sha256sum /etc/ssl/certs/*crt /etc/ssl/certs/*pem /etc/ssl/certs/*0 | grep 58fd6840
# mv /usr/local/share/ca-certificates/myroot.pem /usr/local/share/ca-certificates/myroot.crt
# update-ca-certificates --fresh
[snip]
# sha256sum /etc/ssl/certs/*crt /etc/ssl/certs/*pem /etc/ssl/certs/*0 | grep 58fd6840
58fd68402402917b85858cf61952593a17c8b559991ab969ae9e4dd016472b83 /etc/ssl/certs/myroot.pem
58fd68402402917b85858cf61952593a17c8b559991ab969ae9e4dd016472b83 /etc/ssl/certs/1bc3c8d7.0
# ls -al /etc/ssl/certs/myroot.pem /etc/ssl/certs/1bc3c8d7.0
lrwxrwxrwx 1 root root 10 Jun 16 14:09 /etc/ssl/certs/1bc3c8d7.0 -> myroot.pem
lrwxrwxrwx 1 root root 43 Jun 16 14:09 /etc/ssl/certs/myroot.pem -> /usr/local/share/ca-certificates/myroot.crt
- GitLab charts docs do not specify a filename extension in charts/globals.md: Custom Certificate Authorities
- a
.pem
file is specified for LDAP in charts/globals.md: Using a custom CA or self signed LDAP certificates
Is the filename specified in
kubectl create secret generic custom-ca --from-file=unique_name=/path/to/cert
preserved? Is that's what's written to /usr/local/share/ca-certificates
?
The docs also state:
--set global.certificates.customCAs[0].secret=my-custom-ca.pem
--set global.appConfig.ldap.servers.main.ca_file=/etc/ssl/certs/ca-cert-my-custom-ca.pem
My impression is that Rails uses the hashed filename, not the "pretty" filename, eg:
lrwxrwxrwx 1 root root 10 Jun 16 14:09 /etc/ssl/certs/1bc3c8d7.0
So, if myroot.pem
is there, perhaps to make LDAP work, potentially elements of GitLab will work (Hypothesis: Golang, eg: workhorse, plus LDAP?) but Rails in general will not, because it requires 1bc3c8d7.0
.
Steps to reproduce
(Please provide the steps to reproduce the issue)
Configuration used
(Please provide a sanitized version of the configuration used wrapped in a code block (```yaml))
(Paste sanitized configuration here)
Current behavior
(What you're experiencing happening)
Expected behavior
(What you're expecting to happen)
Versions
- Chart: (tagged version | branch | hash
git rev-parse HEAD
) - Platform:
- Cloud: (GKE | AKS | EKS | ?)
- Self-hosted: (OpenShift | Minikube | Rancher RKE | ?)
- Kubernetes: (
kubectl version
)- Client:
- Server:
- Helm: (
helm version
)- Client:
- Server:
Relevant logs
(Please provide any relevant log snippets you have collected, using code blocks (```) to format)