Draft: PoC Create a pull policy (always-or-fallback) that can fall back on local cache for the docker+machine executor

What does this MR do?

Try to solve the problem specified in #26558 (closed) in a different way. Instead of introducing a new pull policy, update the pull_policy config to support multiple values, for example pull_policy = ["always", "if-not-present"]. This is a PoC to understand the efforts that need to be done to make this happen and to see if this solves the problem our users are having.

Benefits

• Implement it for Kubernetes executor: Before this pull policy was only for Docker/Docker+machine executor, with this we can also implement it for the Kubernetes executor. We can see if the error on the pod is about pulling the image. If it is, we just recreate the pod but this time with the if-not-present policy.
• Simpler to implement: We can just check the error of the pull, if there is an error we log it and move onto the next implementation.
• Not introducing a specific policy for the runner: Not custom/new policy for the runner that other tools and the industry doesn't use.
• Not introducing a new policy with the same security flaws: This is the biggest concern we had when discussing this, there is still the security flaw because we have if-not-present as fallback but at least that is already well documented, and most users are educated around it.
• Stackable: We can stack multiple policies on top of each other, and this might give even more control to what the user might want.
• Less risky to change: We don't have to touch or change any of the existing logic, we just have to wrap it and check if there is an error or not, if there is move onto the next one in line.

Drawbacks

• Breaking change: This of course is going to be a breaking change because from a string of pull_policy = "always" to a slice of strings pull_policy = ["always", "if-not-present"]. We can gradually change this as we did for docker services where we support both syntax for a while and then depreciate it (if we want to remove it). This was done in 52f009a9.

Why was this MR needed?

Allow users to keep using CI even when the registry is done. It keeps the images up to date if the registry is available with always and if pulling fails, it uses the if-not-present policy.

What's the best way to test this MR?

1. docker pull alpine:3.12

2. docker tag alpine:3.12 localonly/alpine:3.12

3. Have the following config.toml

   config.toml

   concurrent = 1
   check_interval = 0
   
   [session_server]
     session_timeout = 1800
   
   [[runners]]
     name = "steve-mbp-gitlab.local"
     url = "https://gitlab.com/"
     token = "5RYH_2sGcGgbmxpq799-"
     executor = "docker"
     [runners.docker]
       tls_verify = false
       image = "localonly/alpine:3.12"
       privileged = false
       disable_entrypoint_overwrite = false
       oom_kill_disable = false
       disable_cache = false
       volumes = ["/cache"]
       pull_policy = ["always", "if-not-present"]
       shm_size = 0

4. Run job, for example https://gitlab.com/steveazz/playground/-/jobs/872091301

What are the relevant issue numbers?

Closes #26558 (closed)