allowed_pull_policies bypass: pull policies not properly filtered on kubernetes and docker executors

This is an issue I've tried to report through hackerone (because of security implications, sounded like the right channel), but it got dismissed (report 2570512).

Trying again here because I don't want to let it unfixed. I'll mostly copy/paste from the H1 report below.

Summary

GitLab Runner executors for Docker and Kubernetes both have an allowed_pull_policies option (see here and here), which restricts which policies may be used for pulling jobs containers images. In both cases, the implementation of this control is flawed, and a CI job can use a pull_policy which is not part of the allowed_pull_policies list, with various security implications: access without credentials to some private images from the container runtime cache on a shared runner, and usage of any image on a runner supposed to be locked on some specific pre-pulled images.

Steps to reproduce

Example for bypassing allowed_pull_policies = ["never"]:

  1. create a group on gitlab.com
  2. follow the group runner registration process (https://gitlab.com/groups/your-group/-/runners/new); choose a distinctive tag (for example docker-never-pull); you get a registration token
  3. launch a runner process using Docker
  4. register the runner on your group, using the token, and choosing the docker executor
  5. edit the /etc/gitlab-runner/config.toml file in your runner container: in the [runners.docker] section, add allowed_pull_policies = ["never"]
  6. restart the runner container to take config change into account
  7. make sure ubuntu:24.04 is not present in your Docker cache (docker image rm ubuntu:24.04)
  8. create a project in the group, with a simple .gitlab-ci.yml file:
stages:
  - test
test:
  stage: test
  tags:
    - docker-never-pull
  image:
    name: ubuntu:24.04
  script:
    - date
  1. check the job is failing (that's expected)
Using Docker executor with image ubuntu:24.04 ...
WARNING: Failed to pull image with policy "never": Error response from daemon: No such image: ubuntu:24.04 (manager.go:163:0s)
ERROR: Preparation failed: failed to pull image "ubuntu:24.04" with specified policies [never]: Error response from daemon: No such image: ubuntu:24.04 (manager.go:163:0s)
  1. try forcing a different pull policy in .gitlab-ci.yml:
...
  image:
    name: ubuntu:24.04
    pull_policy: "if-not-present"
...
  1. check the job is still failing (so far so good, the pull policy restriction is working)
Using Docker executor with image ubuntu:24.04 ...
ERROR: Preparation failed: failed to pull image 'ubuntu:24.04': pull_policy ([if-not-present]) defined in GitLab pipeline config is not one of the allowed_pull_policies ([never])
  1. try again forcing the pull policy in .gitlab-ci.yml (make it a list, with an allowed policy after the forbidden one in the list):
...
  image:
    name: ubuntu:24.04
    pull_policy: ["if-not-present", "never"]
...
  1. this time the job succeeds! You've bypassed the allowed_pull_policies = ["never"] restriction. 
Preparing the "docker" executor 00:09
Using Docker executor with image ubuntu:24.04 ...
Pulling docker image ubuntu:24.04 ...
Using docker image sha256:35a88802559dd2077e584394471ddaa1a2c5bfd16893b829ea57619301eb3908 for ubuntu:24.04 with digest ubuntu@sha256:2e863c44b718727c860746568e1d54afd13b2fa71b160f5cd9058fc436217b30 ...

Note that the effective pull policy which has effectively been used here is the forbidden one (if-not-present), not the allowed one (never, which we've shown would have failed since the image is not in cache).

You can reproduce with a Kubernetes executor. And you can reproduce with a different allowed_pull_policies (["always"] is an other common one), as long as your pull_policy includes a policy which is in allowed_pull_policies. And it works for service:pull_policy too.

Impact

I see two realistic scenarios where bypassing allowed_pull_policies has serious consequences:

  • allowed_pull_policies = ["never"]: attacker is allowed to run jobs on some runners which are dedicated to specific container images (which have been pre-pulled by the runners administrator), and they can actually run any image they want instead (for instance some cryptomining software on some powerful hardware)
  • allowed_pull_policies = ["always"]: the attacker is allowed to run jobs on some shared runners, and the pull policy restriction has been set up to prevent this attack scenario. By bypassing the restriction, attacker can run containers with images they would not have access to otherwise (no registry credentials), but which happen to be in the container runtime cache (because of jobs from other projects using the same shared runners, or other runners on the same infrastructure). These images (private images from other projects sharing the same runners) may contain sensitive data (credentials or signig keys, or restricted software, etc). Some prior knowledge (or guessing) of the images names is required.

Examples

https://gitlab.com/thomasgl-test-runner/test-docker-runner/ is where I've tested the Steps to reproduce; but of course, the Docker runner was on my laptop (and and is now stopped). For Kubernetes, I've been working on a self-managed GitLab server.

What is the current bug behavior?

Docker and Kubernetes executors don't effectively restrict pull_policy based on the allowed_pull_policies option. They try to, but can easily be fooled by pull_policy = ["a-forbidden-policy", "an-allowed-policy"] (a-forbidden-policy will then be tried first). These bugs (twice the same logic mistake) are implemented here:

What is the expected correct behavior?

With allowed_pull_policies = ["an-allowed-policy"], if a job (or service) asks for pull_policy = ["a-forbidden-policy", "an-allowed-policy"], then the job should return an error, as stated in these docs:

The existing pull_policy keyword must not include a pull policy that is not specified in allowed_pull_policies. If it does, the job returns an error.

Edited by Thomas de Grenier de Latour