PF-192: Pipeline job running > hour fails with expired creds

Zach Goldbergrequested to merge
claude-pr/PF-192-31202520184 into main

Implements plan for Linear issue PF-192.

Plan source: https://github.com/gruntwork-io-team/linear-claude-plans/blob/main/plans/PF-192.md

Claude's implementation summary 
Implementation complete across four repos.

## `pipelines-gitlab-init`

- **New `scripts/lib/credentials.sh`** — sourceable library with no side effects on source:
  - `pipelines_mint_gruntwork_read_token <log>` — exchanges `APERTURE_OIDC_TOKEN` for a Gruntwork read token, echoes it on stdout. Writes the node script's output to a `mktemp` file (removed after sourcing) instead of `credentials.sh` in the repo root, and saves/restores `set -x` so debug/trace logging never prints token material.
  - `pipelines_verify_gruntwork_read_token <token> <log>` — the GitHub API probe, returning a status instead of exiting.
  - `pipelines_publish_gruntwork_read_token <token> <source>` — exports the token plus a new `PIPELINES_GRUNTWORK_READ_TOKEN_SOURCE` (`aperture` or `ci_variable`) and rewrites (rather than appends) both keys in `$GITLAB_ENV` and `build.env`. `build.env` is only touched when it already exists, so a refresh in an execute job doesn't drop a file into the customer's checked-out repo.
- **New `scripts/refresh-credentials.sh`** — standalone entrypoint designed to be sourced. No `exec`, no global `set -e`/`set -u`, no `glab` calls, no note collapsing, no `pipelines-gitlab-actions` clone, no CLI install. Skips refresh when the token came from a CI variable or when `APERTURE_OIDC_TOKEN`/`API_BASE_URL` are absent; on a mint failure it falls back to the inherited token (and only hard-fails when there is no token at all). Works both sourced and executed.
- **`scripts/init.sh`** — now sources the library and calls into it, preserving today's precedence (customer CI variable wins and is verified; otherwise mint from Aperture). Tracing is disabled across the whole token section and around the token-bearing `git clone` URL.
- **`scripts/pipelines-credentials.mjs`** — output path now comes from `PIPELINES_CREDENTIALS_OUTPUT_FILE` (default unchanged) and uses `writeFileSync` instead of `appendFileSync`; the inaccurate dotenv comment is replaced.
- **`README.md`** — documents the scripts and the refresh pattern.

Verified with `bash -n`, `shellcheck -x` (clean), `node --check`, and functional smoke tests covering: fresh mint through a sourced call, stale-key replacement in both env files, `ci_variable` skip, mint failure with and without an inherited token, direct execution, and a `set -x` run confirming no token appears in trace output.

## `pipelines`

- `internal/actions/gitlabyml/gitlabdynamicchildren.go` — new `defaultApertureOIDCAudience` const and `resolveApertureOIDCAudience()` (overridable via `APERTURE_OIDC_AUDIENCE`, mirroring `CI_JOB_IMAGE`); `TemplateJobData` gains `ApertureOIDCAudience`.
- `internal/actions/gitlabyml/gitlabinitdynamicchildren.go` — same audience threaded into the reinit template data.
- All six job templates (`execute`, `provision_new_accounts`, `baseline_core_accounts`, `baseline_new_accounts`, `setup_delegated_repo`, `reinit`) now declare an `APERTURE_OIDC_TOKEN` id_token and clone `pipelines-gitlab-init` + source `refresh-credentials.sh` immediately before the `pipelines-gitlab-actions` clone.
- Tests: all nine golden YAML blocks in `gitlabdynamicchildren_test.go` updated; new `TestCustomApertureOIDCAudience`; new `gitlabinitdynamicchildren_test.go` with a golden `TestReinit` (the reinit path previously had no test) and `TestReinitCustomApertureOIDCAudience`.
- `ARCHITECTURE.md` — new cheat-sheet row for the generated GitLab child-pipeline YAML surface.

**`go test` could not be run here**: the module depends on the private `github.com/gruntwork-io-team/pipelines-hooks`, which is neither cached nor fetchable in this environment (no GitHub credentials), so the package fails at `setup failed` before compiling. To compensate I rendered every template through a standalone stdlib-only harness using the identical funcmap and data shapes; this caught and fixed a `{{ ..ApertureOIDCAudience }}` scoping error in three templates, and the golden strings in the tests were produced from that harness's exact byte output. `gofmt` is clean. The Go changes still need a real `go test ./internal/...` run in CI.

## `gitlab-pipelines-workflows`

- New `.refresh_credentials` anchor (clone `pipelines-gitlab-init`, source `refresh-credentials.sh`), used as the first `script:` entry of `status check` ahead of its clone. `Drift Detection` / `Unlock Unit` / `Unlock All` are untouched since they already run `*init_pipelines` first.
- `pipelines_gitlab_init_ref` default bumped `v1.1.3` → `v1.2.0`.
- YAML parses and the anchor expands correctly (nested list, same shape as the existing `*init_pipelines` usage).

**Not done:** `pipelines_binary_version` is still `v0.58.0`. The plan calls for bumping it to the new CLI release, but that version doesn't exist yet and pinning a guessed tag would break the workflow. Per the plan's own release ordering (`pipelines-gitlab-init` tag → CLI release → workflows bump), this pin must be bumped to the actual CLI release before the child-pipeline half of the fix reaches customers. As it stands this PR fixes `status check` only.

## `docs`

- `docs/2.0/docs/pipelines/architecture/security-controls.md` — the GitLab tab of "Token Strategy" now states that Gruntwork GitHub access tokens expire after one hour and are minted per job from the job's OIDC identity, and that a customer-supplied `PIPELINES_GRUNTWORK_READ_TOKEN` CI variable is used as-is and never refreshed.
- `ci-workflows.md` left alone: its GitLab tab doesn't enumerate jobs or steps, so there's nothing to keep in sync.

All repos in the plan's `target_repos` were available; nothing was skipped for a missing repo.

Related PRs in this change set

Edited by Zach Goldberg

Merge request reports