Resource limits not setup for the `init-permissions` container

Status Update (2023-03-22)

You can now apply this configuration using the custom podspec feature. The feature is currently behind a feature flag FF_USE_ADVANCED_POD_SPEC_CONFIGURATION

Config Example

[[runners]]
  environment = ["FF_USE_ADVANCED_POD_SPEC_CONFIGURATION=true"]
  [runners.kubernetes]
    image = "alpine"
    [[runners.kubernetes.pod_spec]]
      name = "tester"
      patch_type = "strategic"
      patch = '''
      initContainers:
        - name: "init-permissions"
          resources:
            limits:
              cpu: "..."
            requests:
              cpu: "..."
      '''

We are also capturing any feedback wrt this feature in the issue Overwrite generated Kubernetes pod specificatio... (#29659 - closed)

Documentation

Status update (2023-02-25)

In 15.10, (2023-03-22), we are working on releasing (behind a feature flag) a new feature that will enable users to pass a PodSpec in either JSON or YAML. This will allow the setting of resource limits for init containers on the runner worker pods. Note - this won't enable setting those limits for the runner manager pod.

Summary

As per this ticket(internal), the customer has set up a security policy to check that all containers have resources limits set. But on setting the limits via the values.yaml file as per below, the init container does not pick up these settings and instead they encounter the error below

ERROR: Job failed (system failure): prepare environment: setting up build pod: admission webhook "validation.gatekeeper.sh" denied the request: [azurepolicy-container-limits-58cd4ab40ea8e76caf7d] container <init-permissions> has no resource limits. Check https://docs.gitlab.com/runner/shells/index.html#shell-profile-loading for more information

Actual behaviour

On investigating further I checked the generated container while a pipeline was running and indeed the resource limits are not set for the init-permissions container

Expected behaviour

It should be possible to configure resource limits for the init-permissions container

Relevant logs and/or screenshots

kubectl describe pod runner-ogbuzdkp-project-2-concurrent-0tsc2c

I described the generated container while a pipeline was running and indeed this is the case

Name:         runner-ogbuzdkp-project-2-concurrent-0tsc2c
Namespace:    gitlab
...
Init Containers:
  init-permissions:
    Container ID:  containerd://491f8f1ae11ffa356d6e4ec52985a452220985f48f8f70d5e0d76ef8da85ed04
    Image:         registry.gitlab.com/gitlab-org/gitlab-runner/gitlab-runner-helper:x86_64-e0218c92
    Image ID:      registry.gitlab.com/gitlab-org/gitlab-runner/gitlab-runner-helper@sha256:7c6b865d5154255db981e1947a3cddf96f64207ef0536658ef10975c8e6162cf
    Port:          <none>
    Host Port:     <none>
    Command:
      sh
      -c
      touch /logs-2-17/output.log && (chmod 777 /logs-2-17/output.log || exit 0)
    State:          Terminated
      Reason:       Completed
      Exit Code:    0
      Started:      Fri, 22 Oct 2021 12:54:22 +0200
      Finished:     Fri, 22 Oct 2021 12:54:22 +0200
    Ready:          True
    Restart Count:  0
    Environment:    <none>
    Mounts:
      /builds from repo (rw)
      /logs-2-17 from logs (rw)
      /scripts-2-17 from scripts (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from default-token-sskf9 (ro)
Containers:
  build:
    Container ID:  containerd://0f128f78b3313a4eaacae97a8a3fb9b7f93cd372daf892451554cc54dad8e379
    Image:         ubuntu:18.04
    Image ID:      docker.io/library/ubuntu@sha256:0fedbd5bd9fb72089c7bbca476949e10593cebed9b1fb9edf5b79dbbacddd7d6
    Port:          <none>
    Host Port:     <none>
    Command:
      sh
      -c
      if [ -x /usr/local/bin/bash ]; then
        exec /usr/local/bin/bash
      elif [ -x /usr/bin/bash ]; then
        exec /usr/bin/bash
      elif [ -x /bin/bash ]; then
        exec /bin/bash
      elif [ -x /usr/local/bin/sh ]; then
        exec /usr/local/bin/sh
      elif [ -x /usr/bin/sh ]; then
        exec /usr/bin/sh
      elif [ -x /bin/sh ]; then
        exec /bin/sh
      elif [ -x /busybox/sh ]; then
        exec /busybox/sh
      else
        echo shell not found
        exit 1
      fi

    State:          Running
      Started:      Fri, 22 Oct 2021 12:54:23 +0200
    Ready:          True
    Restart Count:  0
    Limits:
      cpu:     250m
      memory:  256Mi
    Requests:
      cpu:     100m
      memory:  128Mi

Environment description

config.toml contents

runners:
    locked: false
    config: |
      [[runners]]
        [runners.kubernetes]
        image = "ubuntu:18.04"
        cpu_request = "100m"
        cpu_limit = "250m"
        helper_cpu_request = "100m"
        helper_cpu_limit = "250m"
        service_cpu_request = "100m"
        service_cpu_limit = "250m"
        memory_request = "128Mi"
        memory_limit = "256Mi"
        helper_memory_request = "128Mi"
        helper_memory_limit = "256Mi"
        service_memory_request = "128Mi"
        service_memory_limit = "256Mi"

Used GitLab Runner version

kubectl exec -it gitlab-gitlab-runner-6448ccffb8-x7dk5 -- gitlab-runner --version

Defaulted container "gitlab-gitlab-runner" out of: gitlab-gitlab-runner, configure (init)
Version:      14.3.2
Git revision: e0218c92
Git branch:   14-3-stable
GO version:   go1.13.8
Built:        2021-09-30T16:10:52+0000
OS/Arch:      linux/amd64

Possible fixes

Propagate the resource settings while building the container