### Initial issue description: Unable to upload cache to minio S3 server due to x509 unknown certificate authority I'm running a self-hosted Minio S3 storage server, with a certificate signed by a private authority. Gitlab Runner is running the Kubernetes executor, and has the following cache settings: ```toml [runners.cache] Type = "s3" ServerAddress = "minio.myserver.com" AccessKey = "minio-access-key" SecretKey = "minio-secret-key" BucketName = "gitlab-ci-cache" ``` The tests run fine, however at checking and updating cache, I get the following error: ``` FATAL: Get https://minio.myserver.com/gitlab-ci-cache/runner/b77179e5/project/6/master?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=ZTEPIPIRNBNNCYMSUCYV%2F20180618%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20180618T145218Z&X-Amz-Expires=3600&X-Amz-SignedHeaders=host&X-Amz-Signature=b78a9f63ba05cec3cf6655a50e27549ea0ff6816db50f83ba1316c9c2ce03378: x509: certificate signed by unknown authority ``` Which would be understandable as it is a private certificate authority. However, I have added said CA to the runner's alpine image, both as described in the documentation, and also with update-ca-certificates. That does work for reaching the Gitlab server itself as described, the cache still doesn't work though. I've also added it to the test image (both docker:stable, and docker:stable-dind, as I'm running a docker-in-docker configuration) to no avail. At this point I'm not entirely sure which container communicates with the S3 server, as all of them have the certificate authority installed, yet the error persists. ### Smudge error: Error downloading...x509: certificate signed by unknown authority with LFS remote object store When using remote object store for LFS, the runner gets `Smudge error: Error downloading...x509: certificate signed by unknown authority` when trying to download objects. When the first request - request for a new job - is done, Runner is: - getting the TLS certificate and chain from the GitLab's response, - trying to generate a full certificates chain based on the response data, - attaching resolved chain to the job data, - attaching specified client key/certificate pair (if configured) to the job data. The chain from the job data is next exported as `GIT_SSL_CAINFO` variable during job execution. This becomes the only source of truth for git's certificate validation. If the client key/certificate pair was configured, their content is also exported as `GIT_SSL_KEY` and `GIT_SSL_CERT` variables which allows git to handle TLS Client Authorization that may be configured in front of GitLab. The problem here is that the `GIT_SSL_CAINFO` contains a CA chain that is able to verify only the certificate used by the GitLab instance, from where the job was requested. We've already found this to be problematic for cases when external submodules available through HTTPs are used. Here we have a similar problem caused by LFS. LFS operation is handled by git. Git connects with GitLab (so the certificate at this point is verified properly), but GitLab sends a redirection to S3. Git tries next to connect to S3, but it still is limited by `GIT_SSL_CERT`. Unless the S3 endpoint is signed by a certificate verified by the same CA, this will fail at certificate verification stage. And since they are using Amazon S3, then the domain (and certificate) are *.s3.amazonaws.com. While GitLab's domain is example gitlab.example.com. And this can't be verified. ### Workaround Update the helper image to have the ca chain for the self-signed certificate trusted. ```docker FROM gitlab/gitlab-runner-helper:x86_64-latest RUN apk add --no-cache ca-certificates COPY ca.crt /usr/local/share/ca-certificates/ca.crt RUN update-ca-certificates RUN rm /usr/local/share/ca-certificates/ca.crt ``` `docker build -t registry.gitlab.com/namespace/project/tools/gitlab-runner-helper:$SOME_TAG` [Override](https://docs.gitlab.com/runner/configuration/advanced-configuration.html#overriding-the-helper-image) the helper image used by GitLab by updating the config.toml to use the image you just build with the correct CA trusted. If you are using the helm chart you can define `KUBERNETES_HELPER_CPU_LIMIT` environment variable and define it in [`envVars`](https://gitlab.com/gitlab-org/charts/gitlab-runner/blob/902283d232a5b272603fdce4c11701d43b41fc78/values.yaml#L314-316) ## Proposal NOTE: Mounting a directory and updating the certificates will not work because the helper image is used to create containers multiple types (once per stage). 1. Add an entrypoint to the helper image which will look for a certificate file in well-known location `/etc/gitlab-runner/certs/ca.crt` and install it at every stage. 1. Make sure the entrypoint works as expected. 1. Add a section in the [documentation](https://docs.gitlab.com/runner/configuration/tls-self-signed.html) documenting how to leverage [the entrypoint](https://gitlab.com/gitlab-org/gitlab-runner/blob/3d9a706c59d014409f353da5b1fca1d3197504f0/dockerfiles/alpine/entrypoint#L10-14) to add a certificate to the helper container. Benefits: - This features enables users to leverage a custom CA root certificate in a build script (e.g., to do a secondary Git clone or fetch a file through HTTPS). - This would give 100% control to the user on how certificates should be added to the system since they know their system best. - Users wouldn't have to wait on GitLab to update our integration to add the certificate to the pool.