success of certificates initContainer needs to take into account components of /scripts/bundle-certificates failing
Summary
Rails elements of a customer's Helm deployment of GitLab were not trusting their corporate CA because update-ca-certificates
in /scripts/bundle-certificates
was not completing its job. No hashes of certificates were created.
Customer requested
- we add error handling so failure of
update-ca-certificates
is noisy and visible - we detect that no hashes have been created
More details in:
- A customer issue (ticket for GitLab team members)
- Discussion in related issue #2774 (comment 608458355)
Steps to reproduce
In general, the script won't detect and handle update-ca-certificates
failing.
- script modified so the binary is missing:
/bin # sh -x /scripts/bundle-certificates
+ ls -1 /etc/ssl/certs/
+ wc -l
+ '[' 0 -gt 0 ]
+ update-ca-certificatez
/scripts/bundle-certificates: line 12: update-ca-certificatez: not found
+ readlink -f '/etc/ssl/certs/*.pem'
+ origin='/etc/ssl/certs/*.pem'
+ originPath=/etc/ssl/certs
+ '[' /etc/ssl/certs '!=' /etc/ssl/certs ]
+ readlink -f '/etc/ssl/certs/*.crt'
+ origin='/etc/ssl/certs/*.crt'
+ originPath=/etc/ssl/certs
+ '[' /etc/ssl/certs '!=' /etc/ssl/certs ]
/bin # echo $?
0
- non-executable binary:
/bin # sh -x /scripts/bundle-certificates
+ wc -l
+ ls -1 /etc/ssl/certs/
+ '[' 0 -gt 0 ]
+ update-ca-certificates
/scripts/bundle-certificates: line 12: update-ca-certificates: Permission denied
+ readlink -f '/etc/ssl/certs/*.pem'
+ origin='/etc/ssl/certs/*.pem'
+ originPath=/etc/ssl/certs
+ '[' /etc/ssl/certs '!=' /etc/ssl/certs ]
+ readlink -f '/etc/ssl/certs/*.crt'
+ origin='/etc/ssl/certs/*.crt'
+ originPath=/etc/ssl/certs
+ '[' /etc/ssl/certs '!=' /etc/ssl/certs ]
/bin # echo $?
0
- executes, but fails (binary replaced with a script)
/bin # sh -x /scripts/bundle-certificates
+ ls -1 /etc/ssl/certs/
+ wc -l
+ '[' 0 -gt 0 ]
+ update-ca-certificates
oh no, I failed horribly
+ echo 'the return code was 1'
the return code was 1
+ readlink -f '/etc/ssl/certs/*.pem'
+ origin='/etc/ssl/certs/*.pem'
+ originPath=/etc/ssl/certs
+ '[' /etc/ssl/certs '!=' /etc/ssl/certs ]
+ readlink -f '/etc/ssl/certs/*.crt'
+ origin='/etc/ssl/certs/*.crt'
+ originPath=/etc/ssl/certs
+ '[' /etc/ssl/certs '!=' /etc/ssl/certs ]
/bin # echo $?
0
Configuration used
n/a
Current behavior
It looks likely that update-ca-certificates was segmentation faulting, owing to security software in the customer's environment.
Script does not detect failure of update-ca-certificates
or it having not created any hashes.
It appears that golang doesn't require the hashes, so GitLab trusts the CA to some extent (workhorse, for example) which complicates diagnosing the issue.
Expected behavior
- detect failure of
update-ca-certificates
and ensure script terminates, eg:
update-ca-certificates || exit 1
- measure whether certificate hashes are present.
if update-ca-certificates ; then
if [ 0 -eq $(ls /etc/ssl/certs/*.0 | grep -c ssl/certs) ] ; then
echo "no certificate hashes created"
exit 1
else
:
fi
else
echo "update-ca-certificates failed"
exit 1
fi
Versions
Up to and including chart 5.0.0 (GL14)
Relevant logs
# kubectl logs gitlab-webservice-default-pod -c certificates
Segmentation fault