Fetch instance-level CI/CD variables, and expand user variables in include: entries

Björn Bralarequested to merge
bbrala/glci:91-variables-for-includes into main

Closes #91 (closed), Closes #94 (closed)

Two related changes that came out of running self-managed / DrupalCI pipelines locally.

1. Fetch instance-level CI/CD variables (closes #91 (closed))

glci fetched project and group CI/CD variables from the GitLab API but never instance-level variables, so self-managed instances that rely on them behaved differently from real runners (issue #91 (closed)).

  • Adds FetchInstanceVariables (GET /admin/ci/variables), wired into the daemon's concurrent secret fetch (PreparePipeline) and the resume path.
  • Precedence: instance < group < project (instance is the broadest scope, matching GitLab). Slots into the existing MergeVariables layering and sourceRank.
  • Instance variables require an administrator token. For non-admin tokens GitLab returns 403, which is mapped to an empty set and skipped silently — project/group variables are unaffected.
  • Gated behind --secrets all (the default); --secrets project / none do not fetch them.
  • Cached in the in-memory secure vault alongside project/group vars, and withheld in --secrets project mode just like group vars.

2. Expand user CI/CD variables in include: entries (closes #94 (closed))

GitLab expands $VAR / ${VAR} in include: project: / ref: / file: before fetching the included file (e.g. DrupalCI's project: $_GITLAB_TEMPLATES_REPO, ref: $_GITLAB_TEMPLATES_REF). glci previously only expanded predefined CI_* and the global variables: block there, so includes that referenced user/instance variables failed with HTTP 404 on a literal $VAR.

  • The offline parser, CLI parse path, daemon (PreparePipeline), resume, and child/cross-project pipelines now also make the locally-supplied layers available for include expansion: --env, --env-file, .glci.env, and pipeline-preset env.
  • API-fetched instance/group/project variables are intentionally not used for include expansion — they are fetched concurrently with parsing, so they aren't available when includes resolve. The documented workaround is to supply any variable an include path needs locally (e.g. in .glci.env). This also keeps fetched secrets out of include-fetch URLs.

Docs

  • site/content/user-guide/variables.md: instance-variable precedence/admin-token notes, a new "Variables in include:" section, and a note that child/cross-project pipelines inherit the parent's local layers.
  • site/content/troubleshooting/_index.md: instance-variable troubleshooting entry.
  • site/content/cli-reference/commands.md + .claude/skills/glci/SKILL.md: --secrets all now means project+group+instance; precedence list updated.

Tests

  • pkg/variables/api_test.go: FetchInstanceVariables success + 403 (non-admin) → empty, no error.
  • pkg/variables/resolve_test.go: sourceRank ordering (instance < group < project) and IsSecret for SourceAPIInstance.
  • pkg/variables/secure_vault_test.go: instance-var round-trip in all mode and withholding in project mode.
  • pkg/daemon/executor_test.go: instance layer in MergeVariables; includeExpansionVars excludes API-fetched vars (security property); PreparePipeline never passes API vars to include expansion.
  • pkg/config/fetcher_test.go: variable expansion in include: entries.

Verification

go build ./..., go vet ./..., make test, make test-e2e-parse, and make test-e2e-golden pass. Validated end-to-end against a real Drupal contrib project (jsonapi_extras): with _GITLAB_TEMPLATES_REPO / _GITLAB_TEMPLATES_REF supplied via .glci.env, the DrupalCI templates resolve and the pipeline runs through the daemon execution path (per-matrix images and core versions applied).

Merge request reports