Fix cache keys processing by improving the handling of the fallback keys

What does this MR do?

Fix the runner logic for handling multiple cache keys (primary, fallbacks and global fallback variable) as follows:

• Stop execution after the first successful cache entry has been retrieved
• Allow protected cache keys to be specified in fallback cache keys section

See gitlab!110467 (comment 1364532342) for details

Implementation Details

shells/abstract.go:

• Splits the checks for protected caches so that fallback caches keys are accepted when using protected suffix (not supported for CACHE_FALLBACK_KEY variable)
• Update the logic in addExtractCacheCommand method so that each subsequent check for a cache key happens within the else section of the previous cache key retrieval command (as opposed to the current implementation where all cache keys are being retrieved)

shells/abstract_test.go:

• Consolidate the tests for multiple fallback keys into a single parameterized test

Why was this MR needed?

MR !3875 (merged) with the initial fallback cache keys implementation had the above listed bugs

What's the best way to test this MR?

Instantiate a gitlab instance and configure runners with shared cache

Testing

Setup:

• Spun up a GitLab instance using Gitpod using the branch in MR gitlab!110467 (merged),
• Register shell runner instance using the gitlab-runner implementation on this branch in Gitpod workspace
• Configure shared cache in S3

Job definition:

image: busybox:latest

build1:
  tags:
    - runner
  stage: build
  script:
    - ls -lah
    - cat cached-file.txt || echo "No file found"
    - echo "$(date)" > cached-file.txt
  cache:
    key: latest-cache-3
    paths:
      - cached-file.txt
    fallback_keys:
      - latest-cache-2
      - latest-cache-1

Results:

• latest-cache-3 key is found (latest-cache-2 and latest-cache-1 keys are not retrieved)

• latest-cache-3 key is not found, but latest-cache-2 key is found (latest-cache-1 key is not retrieved)

• latest-cache-3 and latest-cache-2 keys are not found, but latest-cache-1 key is found

What are the relevant issue numbers?

Relates to gitlab#22213 (closed)

Relates to !3875 (merged)

Relates to gitlab!110467 (merged)