Skip to content

Runners with custom ConfigMap & SCC fail chown after Operator v1.37.0 upgrade on OpenShift 4.16.27

Description:

Environment:

  • OpenShift 4.16.27 cluster
  • GitLab Runner Operator v1.37.0 (upgraded on May 16, 2025)
  • Runner custom service account: gitlab-runner-app-sa-chroot (based on anyuid scc)

Configuration Before Update to v1.37.0:

  1. Custom ConfigMap & SCC used to add Linux capabilities for integration with Prisma Twistlock to scan built container images:
  • CAP_SYS_CHROOT
  • CAP_MKNOD
  1. Custom SCC (based on anyuid) granting allowedCapabilities:
  • CAP_SYS_CHROOT
  • CAP_MKNOD
  1. Runners created before May 16, 2025 built concurrent runner pods successfully using the above ConfigMap, SCC, and service account.

What Changed:

  • After upgrading the GitLab Runner Operator to v1.37.0, any runner using a custom ConfigMap began failing at startup with:
 chown: changing ownership of '/builds/.gitlab-build-uid-gid': Operation not permitted

Additional Observation:

  • The Runner Manager Deployment/Pod is configured to use serviceAccountName: gitlab-runner-app-sa-chroot (or gitlab-runner-app-sa by default), but its spawned concurrent runner builder pods show serviceAccountName: default in their spec.
  • Despite this, the builder pods actually execute under the gitlab-runner-app-sa-chroot permissions.
  • To explicitly bind the correct SA on the builder side, the ConfigMap’s config.toml must include:
[runners.kubernetes]
  service_account = "gitlab-runner-app-sa-chroot"

Impact:

  • Only runners that rely on the custom ConfigMap, custom SCC and custom service accounts are affected.
  • Builds now error out during container startup due to the forbidden chown operation, blocking those specific CI runners.

Temporary Workaround:

Adding the following configs to the custom ConfigMap restores functionality:

[runners.kubernetes] 
  cap_add = ["CAP_SYS_CHROOT", "CAP_MKNOD"] 
  service_account_overwrite_allowed = ".*" 
  privileged = false 
  service_account = "gitlab-runner-app-sa-chroot"

  [runners.kubernetes.build_container_security_context] 
    run_as_user = 0 
    run_as_group = 0

  [runners.kubernetes.helper_container_security_context] 
    run_as_user = 0 
    run_as_group = 0

Requested Action:

  1. Investigate why v1.37.0 no longer honors the original CAP_SYS_CHROOT and CAP_MKNOD settings without run_as_user and group to 0.
  2. Propose an alternative workaround that does not require forcing the runner to run as root (UID/GID 0) or binding a specific service account in the TOML. For example:
    • Use a PodSecurityContext with an appropriate fsGroup to allow the container process to write owned files without needing extra capabilities
    • Introduce an init container that performs any necessary chown operations under elevated permissions before the main build container starts

Please let me know if you need additional logs, SCC definitions, or other environment details.