Add wildcard support to cache + recursion detection

What does this MR do and why?

This MR has two commits:

  1. Restore wildcard support in cache keys which broke with MR !203233 (merged). Additionally, this fixes a pre-existing bug in the search_files_by_wildcard_path where **/*.yml patterns failed to match root-level files. This commit is from the MR that had already been reviewed and is the one that got reverted.
  2. Fixes a bug uncovered by another bug that was uncovered by a feature. The wildcard cache fix above made **/*.yml patterns work correctly, but this exposed a bug where a file could accidentally include itself via wildcard. This commit adds additional specs to protect against the bug that caused it.

Screenshots or screen recordings

Commit 2 - Example reproduction (from Slack thread):

#.gitlab-ci.yml
include:
  includes/all.yml

# icludes/all.yml
include:
  - 'includes/**/*.yml'

# includes/d1/job1.yml
job_1:
    script: env

# includes/d2/job2.yml
job_2:
    script: env

Commit 1 - original implementation, still testable this way:

To test different wildcard expansions, you can use the following config:

  test-cache:
    script:
      - echo "Testing cache keys"
    cache:
      - key:
          files: ['*.yml']
          prefix: 'single-wildcard'
        paths: ['vendor/']

      - key:
          files: ['**/*.yml']
          prefix: 'double-wildcard'
        paths: ['vendor/']

      - key:
          files: ['config.yml', '*.json']
          prefix: 'mixed-patterns'
        paths: ['vendor/']

The directory structure you should have is:

  my-project/
  ├── .gitlab-ci.yml         
  ├── config.yml             
  ├── package.json           
  ├── data.json              
  ├── config/
  │   ├── database.yml       
  │   └── settings.json       
  ├── deploy/
  │   └── production.yml      
  └── vendor/                 

Run a pipeline. In the logs, you should see the hash being appended to the caches:

Creating cache double-wildcard-<HASH>...

Without the fix, on master, this would be:

wildcard-default-23-protected

NO HASH

References

#572701 (closed)

MR acceptance checklist

Evaluate this MR against the MR acceptance checklist. It helps you analyze changes to reduce risks in quality, performance, reliability, security, and maintainability.

Edited by Laura Montemayor

Merge request reports

Loading