Skip to content

Allow explicit cpu/memory service resources overwrites for the Kubernetes executor

Tarun Khandelwal requested to merge gitlab-community/gitlab-runner:explicit-service-requests into main

What does this MR do?

Allow explicitly setting resource requirements for services via setting service variables. Example:

  services:
    - name: redis:5
      alias: redis5
      variables:
        KUBERNETES_SERVICE_CPU_REQUEST: "3"
        KUBERNETES_SERVICE_CPU_LIMIT: "6"
        KUBERNETES_SERVICE_MEMORY_REQUEST: "3Gi"
        KUBERNETES_SERVICE_MEMORY_LIMIT: "6Gi"
        KUBERNETES_EPHEMERAL_STORAGE_REQUEST: "2Gi"
        KUBERNETES_EPHEMERAL_STORAGE_LIMIT: "3Gi"
    - name: postgres:12
      alias: MY_relational-database.12
      variables:
        KUBERNETES_CPU_REQUEST: "2"
        KUBERNETES_CPU_LIMIT: "4"
        KUBERNETES_MEMORY_REQUEST: "1Gi"
        KUBERNETES_MEMORY_LIMIT: "2Gi"
        KUBERNETES_EPHEMERAL_STORAGE_REQUEST: "1Gi"
        KUBERNETES_EPHEMERAL_STORAGE_LIMIT: "2Gi"

Why was this MR needed?

While setting uniform service requests/limits for all services within a job works well with similar services having the same needs, precious resources can be wasted if the service topology is heterogeneous: a lightweight Redis service will not need the same resource requests as a MongoDB service. Another database service may be CPU throttled if the global service CPU limits are too restrictive. Still, a user cannot risk changing that global service cpu limit which may lead to node cpu contention if the CPU requests are not precisely tailored.

What's the best way to test this MR?

Although the diff includes the unit tests for the change, I did the sanity testing by doing the following:

  • Created the gitlab-runner image with my changes.
  • Deployed with a helm chart on our cluster connected to a project.
  • Used the below .gitlab-ci.yml and verified the accurate overwrites are applied.

default:
  tags: ['tarun', 'custom-resources-tests']


test-job:
  services:
    - name: postgres:11.7  # custom overwrite
      alias: db
      entrypoint: ["docker-entrypoint.sh"]
      command: ["postgres"]
      variables:
        KUBERNETES_SERVICE_MEMORY_LIMIT: "2Gi"
        KUBERNETES_SERVICE_CPU_LIMIT: "2"
        KUBERNETES_SERVICE_MEMORY_REQUEST: "512Mi"
        KUBERNETES_SERVICE_CPU_REQUEST: "500m"
    
    - name: mysql:latest # default resources from runner
      alias: mysql
      variables:
        MYSQL_DATABASE: $MYSQL_DATABASE
        MYSQL_ROOT_PASSWORD: $MYSQL_ROOT_PASSWORD

  image: alpine:3.7
  script:
    - sleep 1000

build-job:
  image: alpine:3.7
  script:
    - sleep 10000
  services:
    - name: postgres:11.7  # global overwrite
      alias: db
      entrypoint: ["docker-entrypoint.sh"]
      command: ["postgres"]
    - name: mysql:latest  # custom overwrite
      alias: mysql
      variables:
        MYSQL_DATABASE: $MYSQL_DATABASE
        MYSQL_ROOT_PASSWORD: $MYSQL_ROOT_PASSWORD
        KUBERNETES_SERVICE_MEMORY_LIMIT: "2Gi"
        KUBERNETES_SERVICE_CPU_LIMIT: "2"
        KUBERNETES_SERVICE_MEMORY_REQUEST: "512Mi"
        KUBERNETES_SERVICE_CPU_REQUEST: "500m"
  variables:
    KUBERNETES_SERVICE_MEMORY_LIMIT: "3Gi"
    KUBERNETES_SERVICE_CPU_LIMIT: "3"
    KUBERNETES_SERVICE_MEMORY_REQUEST: "1.5Gi"
    KUBERNETES_SERVICE_CPU_REQUEST: "1200m"

What are the relevant issue numbers?

closes #36321 (closed) #25317 (closed)

Also, this diff is based on the changes proposed by @oboukili in !2117 (closed).

Kudos to him!

Edited by Romuald Atchadé

Merge request reports