Docs - product feedback: Gitlab Runner Configurations for Oracle Cloud to support Code Quality Jobs

Gitlab Runner Configurations for Oracle Cloud to support Code Quality Jobs This document outlines specific configurations to be set up for self hosted Gitlab Runners to run code quality jobs in Oracle Cloud Infrastructure.

Problem Statement

GitLab Code Quality jobs running in Oracle Cloud Infrastructure (OCI) environments with Container Runtime Interface (CRI) are experiencing significant performance degradation. These jobs take more than 40 minutes to complete, whereas the same jobs typically execute in approximately 6 minutes when running on alternative cloud providers.

The above problem statement is applicable for the following environment

Cloud Provider: Oracle Cloud Infrastructure (OCI)

Container Runtime: Container Runtime Interface (CRI)

Affected Process: GitLab Code Quality jobs

Gitlab Runners: Self hosted Gitlab Runners

Resolution

The performance issues with Gitlab Code Quality jobs running in OCI with CRI was resolved by

1. Modifying the Gitlab Runner configurations
2. Configure additional parameters in the gitlab-ci file

Modifying the GitLab Runner configurations

Define an empty directory for GitLab Runner Configuration Add the below block in the runners section in values.yaml file. The directory used below will be used to store temp files

    [[runners.kubernetes.volumes.empty_dir]]
      mount_path = "/var/lib"
      name = "docker-data"

Configure additional parameters in the gitlab-ci file

Add DOCKER_DRIVER as empty variable Add the below variable to the existing code quality job in the gitlab-ci file so that default overlay2 is unselected.

DOCKER_DRIVER: ""

Sample runner configuration

The helm chart values.yaml for the gitlab runner which covers the fix mentioned above. (1)

image:
  registry: registry.gitlab.com
  image: gitlab-org/gitlab-runner
  tag: alpine-v16.11.0

useTini: false
imagePullPolicy: IfNotPresent
gitlabUrl: https://gitlab.com/
runnerToken: "<gitlab-runner-token>"
terminationGracePeriodSeconds: 3600
concurrent: 100
shutdown_timeout: 0
checkInterval: 5
logLevel: debug
sessionServer:
  enabled: false
## For RBAC support:
rbac:
  create: true
  rules: []
  clusterWideAccess: false
  podSecurityPolicy:
    enabled: false
    resourceNames:
    - gitlab-runner
metrics:
  enabled: false
  portName: metrics
  port: 9252
  serviceMonitor:
    enabled: false
service:
  enabled: false
  type: ClusterIP
runners:
  config: |
    [[runners]]
      output_limit = 200960
      [runners.kubernetes]
        privileged = true
        allow_privilege_escalation = true
        namespace = "{{.Release.Namespace}}"
        image = "ubuntu:22.04"
        helper_image_flavor = "ubuntu"
        pull_policy = "if-not-present"
        executor = "kubernetes"
        [[runners.kubernetes.volumes.host_path]]
          name = "buildah"
          mount_path = "/var/lib/containers/storage"
          read_only = false
        [runners.kubernetes.volumes]
        [[runners.kubernetes.volumes.empty_dir]]
          mount_path = "/var/lib"
          name = "docker-data"
        [[runners.kubernetes.services]]
          alias = "dind"
          command = [
              "--host=tcp://0.0.0.0:2375",
              "--host=unix://var/run/docker.sock",
          ]
      [runners.cache]
        Type = "s3"
        Path = "gitlab_runner"
        Shared = true
        [runners.cache.s3]
          BucketName = "gitlab-shared-caching"
          BucketLocation = "ap-singapore-1"
          ServerAddress = "<compartmentid>.compat.objectstorage.ap-singapore-1.oraclecloud.com"
          AccessKey = "<Access-key>"
          SecretKey = "<secret_key>"

  configPath: ""
  tags: "<tags-of-your-runner>"
  cache: {}

securityContext:
  allowPrivilegeEscalation: false
  readOnlyRootFilesystem: false
  runAsNonRoot: true
  privileged: false
  capabilities:
    drop: ["ALL"]
strategy: {}
podSecurityContext:
  runAsUser: 100
  fsGroup: 65533
resources: {}
affinity: {}
topologySpreadConstraints: {}
nodeSelector: {}
tolerations: []
## list of hosts and IPs that will be injected into the pod's hosts file
hostAliases: []
deploymentAnnotations: {}
deploymentLabels: {}
podAnnotations: {}
podLabels: {}
priorityClassName: ""
secrets: []
configMaps: {}
volumeMounts: []
volumes: []

Sample Gitlab CI - Code Quality Job Configuration

The gitlab-ci code which covers the fix mentioned above (2)

code_quality: 
services: 
- name: $CODE_QUALITY_DIND_IMAGE command: ['--tls=false', '--host=tcp://0.0.0.0:2375'] 
variables: 
CODECLIMATE_PREFIX: $CI_DEPENDENCY_PROXY_GROUP_IMAGE_PREFIX/
CODECLIMATE_REGISTRY_USERNAME: $CI_DEPENDENCY_PROXY_USER
CODECLIMATE_REGISTRY_PASSWORD: $CI_DEPENDENCY_PROXY_PASSWORD
DOCKER_DRIVER: ""

Note: OCI Best Practices and Procedures

1. To successfully run GitLab pipelines in OCI, it is recommended to use worker node images in OCI K8s nodes.
2. You can use the cache configuration in runners for OCI Object storage since its uses S3 as backend protocol (mentioned in the helm values.yaml file) under runners.cache section