Add wildcard support to cache + recursion detection
What does this MR do and why?
This MR has two commits:
- Restore wildcard support in cache keys which broke with MR !203233 (merged). Additionally, this fixes a pre-existing bug in the
search_files_by_wildcard_pathwhere**/*.ymlpatterns failed to match root-level files. This commit is from the MR that had already been reviewed and is the one that got reverted. - Fixes a bug uncovered by another bug that was uncovered by a feature. The wildcard cache fix above made
**/*.ymlpatterns work correctly, but this exposed a bug where a file could accidentally include itself via wildcard. This commit adds additional specs to protect against the bug that caused it.
Screenshots or screen recordings
Commit 2 - Example reproduction (from Slack thread):
#.gitlab-ci.yml
include:
includes/all.yml
# icludes/all.yml
include:
- 'includes/**/*.yml'
# includes/d1/job1.yml
job_1:
script: env
# includes/d2/job2.yml
job_2:
script: env
Commit 1 - original implementation, still testable this way:
To test different wildcard expansions, you can use the following config:
test-cache:
script:
- echo "Testing cache keys"
cache:
- key:
files: ['*.yml']
prefix: 'single-wildcard'
paths: ['vendor/']
- key:
files: ['**/*.yml']
prefix: 'double-wildcard'
paths: ['vendor/']
- key:
files: ['config.yml', '*.json']
prefix: 'mixed-patterns'
paths: ['vendor/']
The directory structure you should have is:
my-project/
├── .gitlab-ci.yml
├── config.yml
├── package.json
├── data.json
├── config/
│ ├── database.yml
│ └── settings.json
├── deploy/
│ └── production.yml
└── vendor/
Run a pipeline. In the logs, you should see the hash being appended to the caches:
Creating cache double-wildcard-<HASH>...
Without the fix, on master, this would be:
wildcard-default-23-protected
NO HASH
References
MR acceptance checklist
Evaluate this MR against the MR acceptance checklist. It helps you analyze changes to reduce risks in quality, performance, reliability, security, and maintainability.
Edited by Laura Montemayor