feat(glob): Add GitLab's repository wildcard path grammar

Paweł Farysrequested to merge
gitlab-community/gitlab-org/ci-cd/runner-tools/glci:glci-123 into main

What this MR does and why?

include: local: patterns were expanded with filepath.Glob, where ** is just another * that stops at /. So ci/**/*.yml resolved as ci/*/*.yml: it reached neither the file directly under ci/ nor anything two levels down. The pipeline merged fewer files than on GitLab and still reported success, which is the false-green case glci exists to prevent.

GitLab expands these with Repository#search_files_by_wildcard_path, which regex-escapes the whole pattern and then un-escapes exactly three sequences, in order:

regexp_string.gsub!('\*\*\/', '(.*?\/)?')   # any run of directories, or none
regexp_string.gsub!('\*\*',   '.*?')        # any characters, "/" included
regexp_string.gsub!('\*',     '([^\/])*?')  # any characters except "/"

That grammar is now ported into pkg/glob (WildcardPathRegexp) and shared by all three expanders — IncludeResolver.expandLocalGlobs, LocalFetcher.Fetch and resolveLocalIncludes — plus the daemon's trigger-include predicate. Both ends are RE2, since Gitaly compiles GitLab's string with Go's own regexp, so this is an exact port rather than an approximation. Every assertion in spec/models/repository_spec.rb is a test case.

Three behaviour changes come with it, all in the direction of GitLab:

  • ?, [, ], {, } are literal filename characters. GitLab escapes the pattern and un-escapes only *, so ci/[ab].yml names one file rather than matching ci/a.yml and ci/b.yml. containsGlob claimed otherwise, which contradicted both the daemon's own containsGlobMeta (whose comment already cited the upstream reason) and the documented behaviour. The two predicates are now one.
  • Wildcards resolve against the project's non-ignored files. Recursive matching needs a file list rather than a directed descent, so this reuses the enumeration exists: already had. A gitignored fragment is no longer includable — GitLab resolves includes out of the repository and would not see it either.
  • A wildcard no longer expands to the file that declared it, mirroring LocationExpander#expand_wildcard_paths. Without this the ** fix regresses root-level patterns: include: local: '**/*.yml' matches the CI file it is written in, recurses, and trips the depth limit. Confirmed against real GitLab that the same pattern produces a working pipeline (see below), so erroring would have been a divergence introduced by this MR.

Patterns are also no longer filepath.Cleaned — Clean folds ci/**/../x.yml into ci/x.yml and GitLab cleans nothing — and matches stay slash-separated, which incidentally stops includeEntryKey caching one file under two keys on Windows.

pkg/rules had the file-listing walk this needs. It moved to gitutil.ProjectFiles and pkg/rules delegates, rather than the tree carrying a second copy. Its one behaviour change is that the fallback walk is now sorted: WalkDir yields a/b.yml before a.yml, where byte order — git's order, and the order matched includes merge in — puts a.yml first.

Worth flagging for review: the wildcard branch is now contained by construction, because it can only ever return paths that came out of the repository's own file list. The literal branch beside it is unchanged and still resolves lexically, which is #128 — this MR does not widen that, but it does make the asymmetry visible.

Steps to reproduce

mkdir -p repro/ci/one/two && cd repro && git init -q .
printf 'job_a:\n  script: echo a\n' > ci/a.yml
printf 'job_b:\n  script: echo b\n' > ci/one/b.yml
printf 'job_c:\n  script: echo c\n' > ci/one/two/c.yml
printf "include:\n  - local: 'ci/**/*.yml'\nstages: [test]\n" > .gitlab-ci.yml
git add -A && git commit -qm init
glci jobs

Before, only job_b is listed — ** collapsed to one *, so the pattern behaved as ci/*/*.yml. After, all three are.

Full comparison against main, same tree:

Pattern main This MR GitLab
ci/**/*.yml job_b job_a job_b job_c job_a job_b job_c
ci/**.yml job_a job_a job_b job_c job_a job_b job_c
**/*.yml job_a job_a job_b job_c job_a job_b job_c
ci/*.yml job_a job_a job_a
ci/*/*.yml job_b job_b job_b

The GitLab column is not inferred. It was run on a sandbox project: ci/**/*.yml across three depths produced job_a, job_b, job_c, and the root-level **/*.yml — which matches .gitlab-ci.yml itself — produced a working pipeline rather than an error, which is what the self-inclusion skip reproduces.

e2e/testdata/comprehensive gained ci/nested/deep/glob-jobs.yml, included as ci/**/glob-jobs.yml. It sits two directories below ci/, so the old behaviour resolves the pattern as ci/*/glob-jobs.yml and never reaches it — the job is absent and the count assertions fail. No fixture used a wildcard include before, so nothing end to end covered this.

Verified with make test, make test-e2e-parse and make test-e2e-golden. The vendored gitlab-org/gitlab fixture in pkg/config/integration_test.go, which resolves .gitlab/ci/*.gitlab-ci.yml to 35 files, still parses 2471 jobs with 935 overlap.

Relevant issues and other links

Closes #123 (closed)

Merge request reports