[Transient] Inconsistent behavior with pulling runner-helper image when pull_policy is restricted

Summary

There is a test case where we set in runner configuration only to allow never policy with allowed_pull_policies = [never] . Then we would try to apply pull_policy: [always] in CI configuration with expectation that the job would fail with indication that this policy is not allowed.

Most of the time the job would have no problem trying to pull runner-helper image.

Screen_Shot_2022-08-25_at_10.45.49_AM

But sometimes it would failed at trying to pull runner-helper image due to the policy restriction

Screen_Shot_2022-08-25_at_10.19.28

Steps to reproduce

Since this is a transient behavior, the chance of reproducing is low.

  1. Register a runner (might need to enable FF :ci_docker_image_pull_policy if testing happens before the flag is removed)
  2. Update runner's configuration to only allow never policy in /etc/gitlab-runner/config.toml
  3. Create CI yml with pull_policy: [always]
  4. Visit the pipeline and test job (the job should fail) and observe the error message in job log.

Example Project

What is the current bug behavior?

Sometimes the job had no problem pulling runner-helper image, sometimes the pulling attempt was restricted.

What is the expected correct behavior?

I'm not sure, one scenario needs to be decided and should happens consistently - on going discussion https://gitlab.slack.com/archives/CBQ76ND6W/p1661415704071729

Output of checks

Results of GitLab environment info

Expand for output related to GitLab environment info 
(For installations with omnibus-gitlab package run and paste the output of:
`sudo gitlab-rake gitlab:env:info`)

(For installations from source run and paste the output of:
`sudo -u git -H bundle exec rake gitlab:env:info RAILS_ENV=production`)

Results of GitLab application Check

Expand for output related to the GitLab application check 
(For installations with omnibus-gitlab package run and paste the output of:
sudo gitlab-rake gitlab:check SANITIZE=true)


(For installations from source run and paste the output of:
sudo -u git -H bundle exec rake gitlab:check RAILS_ENV=production SANITIZE=true)


(we will only investigate if the tests are passing)

Possible fixes

Edited by Tiffany Rea