Clarify and document best practice for handling use of custom CA by helper to access object storage
In a recent support ticket (ZD internal link) a customer using the Runner Operator in their OpenShift environment was encountering a x509: certificate signed by unknown authority
error in their CI/CD job when the helper tried to store and retrieve cache objects in a separate minio storage system presented behind a certificated signed by their internal custom CA.
In the course of the investigation two options presented themselves to address this:
-
building a custom helper image with the
/etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem
file updated to include the require CA certificate -
creating a secret containing the updated
/etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem
file viaoc create secret generic -n <runner-namespace> tls-ca-bundle --from-file=tls-ca-bundle.pem
and mounting it on the helper (and all other) containers via the followingconfig.toml
configuration:
[runners.kubernetes]
[[runners.kubernetes.volumes.secret]]
name = "tls-ca-bundle"
mount_path = "/etc/pki/ca-trust/extracted/pem"
Neither of these options seems ideal, as the first requires ongoing maintenance to keep the custom image up to date with the GitLab provided image, and the second involves mounting the modified CA bundle file on all containers launched by the runner and keeping the file up to date with any changes made to the default CA bundle.
Attempts involving trying to update the helper CA bundle file after it is launched using pre_build_script
were unsuccessful due to the helper container not running as the root user in OpenShift.
I've raised this issue to ask the following questions:
- are the two options above the current recommended approaches to deal with this situation?
- if so shall we update the general Runner and GitLab Runner Operator documentation accordingly to help deflect support issues?
- should a feature request issue be created to add a configuration option to the Operator to allow a CA cert file to be specified that is automatically incorporated into the helper CA bundle when it is launched, thus avoiding the need for either of the options presented above?
There is some overlap between this issue and gitlab-org/gitlab-runner#27092.