Unexpected cache suffix depending on branch protection

Problem / customer pain:

The CACHE_FALLBACK_KEY no longer useful as caches by default have a suffix attached to their name at time of creation.

Overview

Our cache isn't shared anymore between our default branch (protected) and our feature branches (unprotected) because of a new dynamic suffix added after cache-<index> that depend on ref protection.

---

• <prefix>-<key>-<index>-protected
• <prefix>-<key>-<index>-non_protected

Steps to reproduce

• Use GitLab Caching
• Try to get cache of a protected branch on a non protected branch using FALLBACK_KEY or a lockfile as key

.gitlab-ci.yml

rules:
    - if: $CI_PIPELINE_SOURCE == 'merge_request_event'
      variables:
        CACHE_FALLBACK_KEY: '$CI_JOB_NAME:$CI_MERGE_REQUEST_TARGET_BRANCH_NAME'

  cache:
    key:
      files:
        - yarn.lock
      prefix: project
    paths:
      - .yarn-cache/

  cache:
      key: '$CI_JOB_NAME:$CI_COMMIT_REF_NAME'
      paths:
        - .jestcache/

Actual behavior

A -non_protected suffix is added on the feature branch job:cache:key trying to get the default branch job:cache:key that was created with the -protected suffix. See logs

Expected behavior

A way to delete this suffix.

A new variable could fix it on FALLBACK_KEY but i can't see how to fix it using lockfile as key.

Note that the cache-<index> already causes problems on FALLBACK_KEY and ours depends on $CI_MERGE_REQUEST_TARGET_BRANCH_NAME

Relevant logs and/or screenshots

Default branch job log

Checking cache for jest:mobile:master-3-protected...

Checking cache for project-ea268a7e34ce64f1a07ba41343c9c84cd6a48bb0-3-protected...

Feature branch job log

Checking cache for jest:mobile:1703-link-faq-details-3-non_protected...
WARNING: file does not exist                       
Failed to extract cache
Checking cache for jest:mobile:master...

Checking cache for project-ea268a7e34ce64f1a07ba41343c9c84cd6a48bb0-3-non_protected...

Used GitLab Runner version

Running with gitlab-runner 14.10.0~beta.50.g1f2fe53e (1f2fe53e)
  on blue-4.shared.runners-manager.gitlab.com/default J2nyww-s

Workarounds

• Using lockfile key: generate cache on a non protected branch for each lockfile updates...
• Using fallback key: adding -protected or non_protected manually as cache-<index>...
• gitlab-runner#29033 (comment 928173695)

Potential solutions

• Approach #1: Falling back to the protected cache before hitting CACHE_FALLBACK_KEY:

  scenario	gitlab-runner cache fallback chain
  current scenario	*-non_protected -> $CACHE_FALLBACK_KEY
  alternative scenario	*-non_protected -> *-protected -> $CACHE_FALLBACK_KEY

• Approach #2: Create a scheduled job on a non-protected branch that follows the default branch, and generates a shared cache that is reused by feature branches.

• Approach #3: For self-managed users using remote caches: use a CI job to sync the bucket from the default branch to the current branch.

Proposal

{placeholder for proposal}

moved from gitlab-runner#29033 (moved)

• Sorry if this is a duplicate but i can't find informations on doc / forum / changelogs / issues...
• There is a small relation with gitlab-runner#27293 (moved).
• And it is probably related to the gitlab runner version 14.10.0~beta but i can't find a way to specify another version

It looks like the documentation changes aren't yet live in https://docs.gitlab.com/ee/ci/caching, however there is a follow-up documentation-only MR that will add some more details and is visible here: https://gitlab.com/gitlab-org/gitlab/-/merge_requests/86107/diffs

Thanks, this doc is really clear. It's a regression for us (looking feature) because non_protected only pull cache from protected without push but i saw your technical / security reasons, we'll try to do otherwise

Introduced in GitLab 15.0.

@pedropombeiro I see that the doc mention it's introduced with 15.0, but the change have been back-ported in 14.8.6, 14.9.4 and 14.10.1. Also the link to the issue is incorrect.

I see that the doc mention it's introduced with 15.0, but the change have been back-ported in 14.8.6, 14.9.4 and 14.10.1.

@tchandelle The documentation is correct: the change was introduced in %15.0, but per our security release backporting documentation it gets backported to some previous monthly releases. This can be seen in the blog post for the latest security releases:

Also the link to the issue is incorrect.

Unfortunately, we're not ready to disclose additional information for the time being to allow users enough time to safely upgrade, so that merge request needs to be kept private.

@tchandelle Also, the link for "Introduced" word I think it's broken, I receive 404 when opening that #330047 (closed)

@francovp it is not broken, the issue is just currently marked as Confidential for security reasons.

changed the description

Same problem has just bitten us. We use the master branch unit test running times to parallelise the unit tests in the feature branches and now we can't do that and all our feature pipelines are failing.

Same here - our cache is getting stored in minio with a "-non-protected" suffix to the name. A subsequent job to delete these is now failing as the name is not what i expects.

We're on v14.9.1 of gitlab runner still.

Same here

We are using v13.11.0 of gitLab-runner

Ok, so version of the runner doesn't seem to matter - it must be a backend itlab.com change that happened today.

That is correct, this was part of a security fix. The idea was that caches would be regenerated under the new names, making it a transparent operation for users - other than a one-time performance hit while regenerating the cache. I see now that this was likely an optimistic view of the problem, considering the myriad ways that users may be leveraging caches.

That being said, it is important for us to understand what your different scenarios are, that have prevented things from going as planned, so I would really appreciate it if users coming here could explain why regenerating the cache under a new name is breaking their workflow.

More scared than hurt for us

Use cases

1. MR Pipelines - Main lockfile cache

Lockfile cache (based on yarn.lock) started to fail on MR Pipelines from non protected source branches. Why? The job generating this cache is called only on specific changes (maybe a bad practice?) and other jobs are only pulling this cache.

Generating a new lockfile cache for non_protected branches (just once) FIXED the problem, we won't need to do this manipulation again, future lockfile keys will be correctly generated.

The only difference for me is that there are lockfile caches for protected branches and lockfile caches for non protected branches. It is not a real problem but this "duplicate" will of course add some CI minutes in our process.

2. MR Pipelines - QA / Tests cache

In order to use QA/Tests Widgets/Reports & Badges with correct values, we have to lint / test our entire project (not only changes). We're using cache to save CI minutes.

But on MR Pipelines we use $CI_JOB_NAME:$CI_MERGE_REQUEST_TARGET_BRANCH_NAME as FALLBACK_KEY in order to pull job:cache from the target branch on the 1st pipeline saving CI minutes and then generating job:cache for the source branch using $CI_JOB_NAME:$CI_COMMIT_REF_NAME as job:cache:key

This is adding some CI minutes again because FALLBACK_KEY is not working anymore (note that we were already impacted by the cache<index>) but this won't affect our pipelines status, it's ok.

---

Maybe we're doing wrong ? But the main problem for me is FALLBACK_KEY, what's its purpose? It doesn't include any cache key suffix but i can manually specify <key>-3-protected ? Moreover, isn't it just a fallback that only check/pull accessible cache without any security issue ?

Edit: you've just answered here

We've worked around the problem by using a provider bucket (in our case AWS). e.g.

  script:
    - ./scripts/make-aws-config-file.sh
    - aws s3 sync s3://<secret-bucket-name>/master/reports reports/

changed the description

added [Deprecated] Category:Runner devopsverify grouprunner labels

added sectionops label

marked this issue as related to #330047 (closed)

cc @DarrenEastman @erushton @dcouture

mentioned in issue #330047 (closed)

Hello ! This affects my teams with no satisfying workaround.

Use case : Gitlab Pages does not provide a per-branch HTML serving. We are using cache to gather previously generated HTML from other branches, updating current branch content, and then serving all branches while updating cache for next run.

Now master cache is not accessible anymore. We will probably unprotect our master branch for now, which is (very) far from ideal. I don't see how CACHE_FALLBACK_KEY can help me, since unprotected cache is available, hence fallback one is not accessed.