Skip to content

GitLab-CI fails to cache to Minio with selfsigned certificate

Summary

I have been setting a GitLab instance up on EKS with selfsigned certificates and a company Root CA, almost everything seems to work, but when i try to use gitlab-ci cache pushing to minio it fails.

Steps to reproduce

Use helm to deploy everything with selfsigned certificates, and custom Root CA

Configuration used

(Please provide a sanitized version of the configuration used wrapped in a code block (```yaml))

gitlab:
  nginx-ingress:
    controller:
      service:
        externalTrafficPolicy: "Cluster"
        annotations:
          service.beta.kubernetes.io/aws-load-balancer-internal: "true"
          service.beta.kubernetes.io/aws-load-balancer-backend-protocol: tcp
          service.beta.kubernetes.io/aws-load-balancer-ssl-ports: "443"
          service.beta.kubernetes.io/aws-load-balancer-ssl-cert: "arn:aws:acm:****"
          service.beta.kubernetes.io/aws-load-balancer-backend-protocol: "https"
  global:
    hosts:
      domain: jackpot.domain.example
      gitlab:
        name: gitlab.jackpot.domain.example
    appConfig:
      ldap:
        servers:
          main:
            label: LDAP Login
            host: ldap.domain.example
            port: 389
            uid: sAMAccountName
            password:
              secret: ldap-main-password
            bind_dn: "***"
            verify_certificates: false
            active_directory: true
            base: DC=root,DC=domain,DC=example
    certificates:
      customCAs:
        - secret: gitlab-certificate-ca-company
        - secret: gitlab-runner-ca-crt
        - secret: gitlab-certificate-ca
  ingress:
    configureCertmanager: false
  certmanager-issuer:
    email: me@example.com
  gitlab-runner:
    imagePullPolicy: Always
    runnerRegistrationToken: "qiaqCncm1ppfkF9HGI2qwDNhHoGsVe5oRPlpeJe9IbndGgjUMmCB34q7akL1c47a"
    certsSecretName: "gitlab-runner-ca-crt"
    concurrent: 30
    requestConcurrency: 30
    checkInterval: 3
    rbac:
      create: false
      clusterWideAccess: true
    imagePullSecrets:
      - regcred
    runners:
      image: ubuntu:18.04
      privileged: true
      namespace: gitlab-runner
      cache:
        ## General settings
        cacheType: s3
        cacheShared: true
        s3ServerAddress: minio.jackpot.domain.example
        s3CacheInsecure: false
        secretName: jackpot-play-minio-secret
        
certmanager:
  install: false

Current behavior

Currently upon trying to cache the system i get the following error when trying to get the cache

Checking cache for ci-cd...
WARNING: Retrying...                                error=Get https://minio.jackpot.domain.example/runner-cache/gitlab_runner/project/15/ci-cd?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=[FILTERED]&X-Amz-Date=20190830T091032Z&X-Amz-Expires=3600&X-Amz-SignedHeaders=host&X-Amz-Signature=[FILTERED]: x509: certificate signed by unknown authority
WARNING: Retrying...                                error=Get https://minio.jackpot.domain.example/runner-cache/gitlab_runner/project/15/ci-cd?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=[FILTERED]&X-Amz-Expires=3600&X-Amz-SignedHeaders=host&X-Amz-Signature=[FILTERED]: x509: certificate signed by unknown authority
FATAL: Get https://minio.jackpot.domain.example/runner-cache/gitlab_runner/project/15/ci-cd?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=[FILTERED]&X-Amz-Date=20190830T091032Z&X-Amz-Expires=3600&X-Amz-SignedHeaders=host&X-Amz-Signature=[FILTERED] x509: certificate signed by unknown authority

and upon trying to store the data:

Creating cache ci-cd...
renderer/node_modules/: found 41406 matching files 
renderer/public/: found 4 matching files           
WARNING: renderer/vendor/: no matching files       
Uploading cache.zip to https://minio.jackpot.domain.example/runner-cache/gitlab_runner/project/15/ci-cd 
WARNING: Retrying...                                error=Put https://minio.jackpot.domain.example/runner-cache/gitlab_runner/project/15/ci-cd?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=[FILTERED]&X-Amz-Date=20190829T074407Z&X-Amz-Expires=3600&X-Amz-SignedHeaders=host&X-Amz-Signature=[FILTERED]: x509: certificate signed by unknown authority
Uploading cache.zip to https://minio.jackpot.domain.example/runner-cache/gitlab_runner/project/15/ci-cd 
WARNING: Retrying...                                error=Put https://minio.jackpot.domain.example/runner-cache/gitlab_runner/project/15/ci-cd?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=[FILTERED]&X-Amz-Date=20190829T074407Z&X-Amz-Expires=3600&X-Amz-SignedHeaders=host&X-Amz-Signature=[FILTERED]: x509: certificate signed by unknown authority
Uploading cache.zip to https://minio.jackpot.domain.example/runner-cache/gitlab_runner/project/15/ci-cd 
FATAL: Put https://minio.jackpot.domain.example/runner-cache/gitlab_runner/project/15/ci-cd?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=[FILTERED]&X-Amz-Date=20190829T074407Z&X-Amz-Expires=3600&X-Amz-SignedHeaders=host&X-Amz-Signature=[FILTERED] x509: certificate signed by unknown authority 

Expected behavior

loading and storing the cache as its supposed to

Versions

  • Chart: 12.1.2
  • Platform:
    • Cloud: EKS
  • Kubernetes: (kubectl version)
    • Client: Client Version: version.Info{Major:"1", Minor:"15", GitVersion:"v1.15.3", GitCommit:"2d3c76f9091b6bec110a5e63777c332469e0cba2", GitTreeState:"clean", BuildDate:"2019-08-19T12:36:28Z", GoVersion:"go1.12.9", Compiler:"gc", Platform:"darwin/amd64"}
    • Server: Server Version: version.Info{Major:"1", Minor:"13+", GitVersion:"v1.13.8-eks-a977ba", GitCommit:"a977bab148535ec195f12edc8720913c7b943f9c", GitTreeState:"clean", BuildDate:"2019-07-29T20:47:04Z", GoVersion:"go1.11.5", Compiler:"gc", Platform:"linux/amd64"}
  • Helm: (helm version)
    • Client: Client: &version.Version{SemVer:"v2.14.3", GitCommit:"0e7f3b6637f7af8fcfddb3d2941fcc7cbebb0085", GitTreeState:"clean"}
    • Server: Server: &version.Version{SemVer:"v2.14.2", GitCommit:"a8b13cc5ab6a7dbef0a58f5061bcc7c0c61598e7", GitTreeState:"clean"}

Relevant logs

Unknown

To upload designs, you'll need to enable LFS and have an admin enable hashed storage. More information