Unexpected cache suffix depending on branch protection

Problem / customer pain:

The CACHE_FALLBACK_KEY no longer useful as caches by default have a suffix attached to their name at time of creation.

Overview

Our cache isn't shared anymore between our default branch (protected) and our feature branches (unprotected) because of a new dynamic suffix added after cache-<index> that depend on ref protection.

---

• <prefix>-<key>-<index>-protected
• <prefix>-<key>-<index>-non_protected

Steps to reproduce

• Use GitLab Caching
• Try to get cache of a protected branch on a non protected branch using FALLBACK_KEY or a lockfile as key ❌

.gitlab-ci.yml

rules:
    - if: $CI_PIPELINE_SOURCE == 'merge_request_event'
      variables:
        CACHE_FALLBACK_KEY: '$CI_JOB_NAME:$CI_MERGE_REQUEST_TARGET_BRANCH_NAME'

  cache:
    key:
      files:
        - yarn.lock
      prefix: project
    paths:
      - .yarn-cache/

  cache:
      key: '$CI_JOB_NAME:$CI_COMMIT_REF_NAME'
      paths:
        - .jestcache/

Actual behavior

A -non_protected suffix is added on the feature branch job:cache:key trying to get the default branch job:cache:key that was created with the -protected suffix. See logs

Expected behavior

A way to delete this suffix.

A new variable could fix it on FALLBACK_KEY but i can't see how to fix it using lockfile as key.

Note that the cache-<index> already causes problems on FALLBACK_KEY and ours depends on $CI_MERGE_REQUEST_TARGET_BRANCH_NAME

Relevant logs and/or screenshots

Default branch job log

Checking cache for jest:mobile:master-3-protected...

Checking cache for project-ea268a7e34ce64f1a07ba41343c9c84cd6a48bb0-3-protected...

Feature branch job log

Checking cache for jest:mobile:1703-link-faq-details-3-non_protected...
WARNING: file does not exist                       
Failed to extract cache
Checking cache for jest:mobile:master...

Checking cache for project-ea268a7e34ce64f1a07ba41343c9c84cd6a48bb0-3-non_protected...

Used GitLab Runner version

Running with gitlab-runner 14.10.0~beta.50.g1f2fe53e (1f2fe53e)
  on blue-4.shared.runners-manager.gitlab.com/default J2nyww-s

Workarounds

• Using lockfile key: generate cache on a non protected branch for each lockfile updates...
• Using fallback key: adding -protected or non_protected manually as cache-<index>...
• gitlab-runner#29033 (comment 928173695)

Potential solutions

• Approach #1: Falling back to the protected cache before hitting CACHE_FALLBACK_KEY:

  scenario	gitlab-runner cache fallback chain
  current scenario	*-non_protected -> $CACHE_FALLBACK_KEY
  alternative scenario	*-non_protected -> *-protected -> $CACHE_FALLBACK_KEY

• Approach #2: Create a scheduled job on a non-protected branch that follows the default branch, and generates a shared cache that is reused by feature branches.

• Approach #3: For self-managed users using remote caches: use a CI job to sync the bucket from the default branch to the current branch.

Proposal

{placeholder for proposal}