Skip to content

Distributed Cache Not Working with S3 Compatible Ceph Object Gateway

Summary

I'm trying to setup Distributed Cache in my Gitlab Runner in order to speed up the execution of Docker Build. For this I plan to utilise Ceph Object Gateway which is a S3 compatible Object Store like MiniO. However even after updating information related to Ceph, Gitlab Runner doesn't seem to be utilising it at all. I'm not seeing anything in logs wherein it even tries to utilise Ceph to store anything.

Steps to reproduce

Test Gitlab Runner distributed cache on Ceph Object Gateway (Version - Octopus 15.2.8)

Gitlab Runner Helm Chart values.yaml


gitlabUrl: https://10.10.0.0/

runnerRegistrationToken: "xxxxx"

unregisterRunners: true

terminationGracePeriodSeconds: 3600

certsSecretName: 10.10.0.0-cert

concurrent: 10

checkInterval: 30

rbac:
  create: true
  clusterWideAccess: false
  podSecurityPolicy:
    enabled: false
    resourceNames:
    - gitlab-runner

metrics:
  enabled: true

runners:
  config: |
    [[runners]]
      name = "Kubernetes Runner"
      executor = "kubernetes"
      [runners.kubernetes]
        image = "alpine:latest"
        privileged = true
        cpu_request = "6"
        cpu_limit = "10"
        memory_request = "6Gi"
        memory_limit = "10Gi"
        ephemeral_storage_request = "10Gi"
        ephemeral_storage_limit = "20Gi"
        helper_cpu_request = "6"
        helper_cpu_limit = "10"
        helper_memory_request = "6Gi"
        helper_memory_limit = "10Gi"
        helper_ephemeral_storage_request = "10Gi"
        helper_ephemeral_storage_limit = "20Gi"
        service_cpu_request = "6"
        service_cpu_limit = "10"
        service_memory_request = "6Gi"
        service_memory_limit = "10Gi"
        service_ephemeral_storage_request = "10Gi"
        service_ephemeral_storage_limit = "20Gi"
        poll_interval = 5
        poll_timeout = 3600
        dns_policy = "default"
        [dns_config]
          nameservers = [
            "10.10.0.1",
          ]
      
      [runners.cache]
        Type = "s3"
        Path = "/cache/"
        Shared = true
        [runners.cache.s3]
          ServerAddress = "http://10.10.0.2:8080"
          AccessKey = "xxxxx"
          SecretKey = "xxxxx"
          BucketName = "gitlab-runner"
          BucketLocation = ""
          Insecure = true

      [[runners.kubernetes.volumes.empty_dir]]
        name = "docker-certs"
        mount_path = "/certs/client"
        medium = "Memory"

  tags: "telco-cdl-gitlab-runner"
  name: "telco-cdl-gitlab-runner"

securityContext:
  fsGroup: 65533
  runAsUser: 100

Environment description

I'm using Shared Runner at Group level, running using Kubernetes Executor on our K8s Cluster (On-Premise)

Used GitLab Runner version

Running with gitlab-runner 13.8.0 
Preparing the "kubernetes" executor
Using Kubernetes namespace: gitlab-runner
Edited by Prateek Dubey