ci: ephemeral-first pipelines and a human-gated release (!7) · Merge requests · GitLab.com / Public Sector / STIG Analyzer

What does this MR do?

Makes this project's own pipeline mean something. Three files change: .gitlab-ci.yml, CHANGELOG.md, and STIGMAN_DEPLOYMENT.md. Nothing else.

Four things happen here:

The AWS/Lightsail lane this project ran on itself is removed -- a STIG Manager we operate persistently has no sustaining value.
Merge request pipelines now run real jobs on a CI-only diff. Before this MR, a diff that rewrote the pipeline ran no build and no tests, so a CI change could not be reviewed against evidence -- including this MR's own.
The release lane is corrected and human-gated. It computed a version below published history, claimed a catalog publication it never performed, and would have minted a tag on every master merge with nobody in the loop.
The Duo lane stays fail-loudly and stays red. It is red because the configured token holds no Duo entitlement, not because of anything CI can fix. This MR does not hide that.

Scope boundary: templates/stig-compliance-analysis.yml is a consumer-facing component and is byte-identical to master after this MR. Every removal below is scoped to jobs this project runs on itself.

This is a redraft. An adversarial review of the first attempt found the Duo gate was a no-op, the version math wrong, and one "fix" a net regression. What the review changed lists each reversal.

1. AWS/Lightsail lane: removed, not gated

Removed from .gitlab-ci.yml:

Anchors .aws_base, .aws_auto_rules, .aws_manual_rules, .aws_persist_rules.
Jobs stig::stigmanager::provision, ::deploy, ::stop, ::reconcile, ::upload.
The deploy stage from stages: -- nothing else used it.

::upload goes too, not merely orphaned: its only non-fallback needs: source was ::deploy's dotenv (STIGMAN_URL/AUTH_URL), which no longer exists. Its replacements are both consumer-facing and neither is operated by us: a follow-up ephemeral in-pipeline import-verify job (containers as services: import, assert, artifact, die), and a consumer-pointed push via the template's own inputs.

build_deployer_image stays. It builds aws/Dockerfile, but this project's own AWS jobs never consumed that image -- .aws_base called aws/lightsail-deploy.sh directly via amazon/aws-cli. The deployer image's only consumer is the template's consumer-facing stigmanager jobs.

2. `aws/` stays whole; `STIGMAN_DEPLOYMENT.md` is rewritten

No file under aws/ is removed, and none of it is dead. The first draft called aws/lightsail-deploy.sh a follow-up removal candidate. That was wrong: aws/Dockerfile:33 copies it to /usr/local/bin/lightsail-deploy.sh, and templates/stig-compliance-analysis.yml invokes it by bare command name off the deployer image's PATH (lightsail-deploy.sh provision / deploy / stop, lines 503/512/538). Deleting it would break the consumer feature.

STIGMAN_DEPLOYMENT.md is rewritten, and only where it described jobs this project no longer runs:

A header note saying plainly that this project runs no STIG Manager of its own and that the stig::stigmanager::* jobs live in the component template, instantiated in a consumer's pipeline via deploy_stigman: true.
The lifecycle table is labelled as the template's jobs. Its ::upload row no longer claims a .ckl arrives from test_aggregator (a job in this project's pipeline, which nothing consumer-facing can reach); it names stig::analyze and the child pipeline's stig::aggregate, which is what the template's own job actually does.
The CI/CD-variable section stops telling readers to configure AWS_ROLE_ARN and LIGHTSAIL_SSH_KEY. Those unprefixed names were read only by the removed in-repo job; they are now marked as used by nothing.
The demo walkthrough is addressed to a consumer project.

The AWS/Lightsail mechanics themselves (OIDC->STS, snapshots, Caddy/sslip.io, cost table, hardening) are untouched.

3. No Duo presence gate; the Duo lane is expected red

The first draft added .full_pipeline_rules_duo_gated -- .full_pipeline_rules with every branch ANDed against $GITLAB_DUO_TOKEN -- on the theory that a merge request pipeline never receives the variable. That anchor is deleted. Every job uses .full_pipeline_rules. Two reasons, either sufficient:

GITLAB_DUO_TOKEN is an unprotected (masked + hidden) project variable with environment scope *. It is therefore defined at pipeline-creation time on every ref, MR pipelines included. The gate collapsed into an exact duplicate of .full_pipeline_rules: 24 lines that changed nothing.
Presence was never the failure mode. The configured token is a project-bot token with api scope. It authenticates; the Duo Agent Platform then refuses it (AgenticChatForbiddenError) because the bot identity holds no Duo entitlement. No pipeline-creation-time check can detect that.

So stig::verify-creds keeps fail-loudly semantics -- no allow_failure, no gate -- and is expected red on every ref until an entitled token exists. That red is the standing reminder that the entitlement is missing. The blast radius is bounded: only the eight jobs behind it are blocked -- the seven with a direct needs: stig::verify-creds (test_ai_connectivity, test_with_demo_app, test_gkg_duo_mcp, test_batch_1/2/3, test_parallel_with_limit), plus test_aggregator behind the batch jobs. Build, the four non-Duo tests, duo_cli_canary, container scanning and the release lane all complete independently (see pipeline shapes).

This MR does not claim master goes green. It claims master stops carrying reds that CI itself caused.

4. CI-touching diffs now run the real build and test lane

.full_pipeline_rules' changes: list matched only stig_tools/**/*. So a merge request that rewrote .gitlab-ci.yml, the component template, or scripts/ got no build and no tests -- only SAST, secret detection and dependency scanning. A CI change could not be reviewed against evidence.

The list now also covers .gitlab-ci.yml, templates/**/* and scripts/**/*. container_scanning's hand-copied rules share the identical list through a YAML anchor (&ci_relevant_paths), so the two cannot drift on paths; only the six if: conditions remain hand-copied, and the in-file comment says so.

This MR's own pipeline is the first beneficiary: it runs the full build + test lane, container scanning against the real image, and a compute_version dry run. It will be red at stig::verify-creds, for the entitlement reason in section 3.

5. `container_scanning`: point it at the real image, keep the kill switch

CS_IMAGE becomes $ANALYZER_IMAGE:$CI_COMMIT_SHORT_SHA -- the image build_analyzer_image actually produces -- with needs: build_analyzer_image. variables: deep-merges with the template's GIT_STRATEGY: none; rules: is an array, so redefining it replaces the template's outright.

The override keeps the template's own kill switch as rule 1, verbatim:

- if: '$CONTAINER_SCANNING_DISABLED == "true" || $CONTAINER_SCANNING_DISABLED == "1"'
  when: never

An override that replaces rules: wholesale must not silently drop the escape hatch the template ships. Rules 2-7 are the six .full_pipeline_rules conditions, copied by hand because a YAML merge key resolves per mapping key: a job that declares its own rules: cannot also pull in an anchor's, so there is no way to prepend a rule to a <<:-merged array.

6. `compute_version`: it was minting a version below published history

The old logic took the highest git tag and emitted a bare, unprefixed tag. Both wrong:

Real tags stop at v1.0.6, but the absorbed CHANGELOG documents [1.2.4] as released. Tag-only math emits 1.0.7 -- below published history.
This project's tags are v-prefixed. A bare tag starts a second scheme.

New rule, deliberately simple and commented in place:

Take the highest released version across both the v* git tags and the CHANGELOG's released ## [X.Y.Z] headings.
Bump MINOR when ## [Unreleased] carries an ### Added or ### Changed subsection; otherwise PATCH.
Mint a v-prefixed tag.
Fail (exit 1) if no released version is found anywhere -- the old code silently defaulted to 1.0.0. That diagnostic is now actually reachable: under set -o pipefail a grep matching nothing killed the command substitution before the emptiness check could run, so the branch was dead code. A || true on the substitution restores it.

RELEASE_VERSION still overrides, still takes bare MAJOR.MINOR.PATCH, and now gets the v added for it -- but it is rejected (exit 1) unless it is strictly greater than the highest released version across tags and CHANGELOG. Re-using or lowering the current version would publish a release behind history under a fresh tag.

Executed against a full-fetch clone of this repository with this MR's CHANGELOG in place (proof): v1.3.0.

release's needs: is [compute_version, build_analyzer_image, container_scanning], so the manual button cannot light up before the artifact under review has been built and scanned (container_scanning is allow_failure: true, so it gates on completion, not on a clean scan). The test stage is deliberately not in that list: gating on it would re-couple releasing to the Duo entitlement. So test-stage state is visible on the pipeline but unenforced at click time -- judging the whole pipeline before clicking is the human's job, which is the point of the manual gate.

compute_version also gains needs: []. Not cosmetic: without it the release stage is stage-ordered behind the test stage, and the knowingly-red stig::verify-creds would make the release lane unreachable for as long as the Duo entitlement is missing -- coupling "can we ship" to "does the bot have a Duo seat." As a DAG job it starts immediately. The human clicking release still sees the whole pipeline's state.

It now also runs as a dry run on merge requests that touch .gitlab-ci.yml or CHANGELOG.md -- the two inputs it reads -- so the math is exercised by the MR that changes it. Nothing is minted there; release is default-branch-only.

7. `release_to_catalog` -> `release`, and minting is human-gated

Renamed, because it does not publish to the CI/CD Catalog and never did. --publish-to-catalog applies only to a project marked as a catalog resource, and this project is not one (isCatalogResource: false). The old name claimed otherwise in the job name, its comments, and the previous MR description. Whether to mark this project a catalog resource is a separate pending decision and nothing in this MR depends on it.

Human-gated: when: manual on the default branch, so no tag is ever minted without a human clicking. allow_failure: true keeps it an optional manual job, so an un-clicked master pipeline reports success rather than sitting blocked forever.

The first click will mint v1.3.0 and publish CHANGELOG.md as the release description. That is a conscious decision at click time, not a merge-time surprise.

8. Duplicate-tag guard: no `GITLAB_TOKEN`, and a three-way exit code

git ls-remote --exit-code returns three distinct things, verified in the release job's actual image (registry.gitlab.com/gitlab-org/cli:latest -- Alpine, git 2.54.0): 0 the tag exists, 2 the tag is free, 128 the call failed (transport, DNS, auth). A single if git ls-remote ...; then collapses 2 and 128 into one branch, so a registry outage reads as "the tag is free" and the job mints blind. The guard therefore branches explicitly, and anything unrecognised refuses:

rc=0
git ls-remote --exit-code --tags origin "refs/tags/$NEW_TAG" >/dev/null 2>&1 || rc=$?
case "$rc" in
  0) echo "ERROR: tag $NEW_TAG already exists on origin ..."; exit 1 ;;
  2) echo "Tag $NEW_TAG is free on origin." ;;
  *) echo "ERROR: git ls-remote failed (rc=$rc) ... Refusing to mint blind."; exit 1 ;;
esac

No credential handling is needed: the runner already points origin at CI_REPOSITORY_URL with the job token.

The first draft's fix -- variables: GITLAB_TOKEN: $CI_JOB_TOKEN -- is removed. It was worse than the bug it fixed:

glab resolves GITLAB_TOKEN first and sends it as a PRIVATE-TOKEN header. The job-token allowlist honours only JOB-TOKEN / token= / job_token= forms, so the guard would have 401'd.
The same variable overrides the stored job token inside the glab release create login GitLab generates for release: -- poisoning the release step itself. glab's own help warns about exactly this.

Honest state

	Before this MR (master)	After this MR
MR pipeline on a CI-only diff	no build, no tests -- `duo_cli_canary`, the 5 stigmanager jobs, template scanners	17 jobs: build, all 14 test-stage jobs, real container scan, `compute_version` dry run
`container_scanning`	runs on every ref, fails pulling a path this project never builds	scans the image this pipeline built; `allow_failure: true`, so a finding does not redden the pipeline
Duo lane (`stig::verify-creds` + 8 dependents)	red -- token refused for lack of entitlement	still red, same reason. Not gated, not `allow_failure`. Unblocked only by an entitled token
Release lane on master	`compute_version` -> `release_to_catalog` would fire once earlier stages passed; would mint `1.0.7` (bare) and claim a catalog publish	`compute_version` runs; `release` is a manual button. First click mints `v1.3.0`
AWS lane on master	`provision`/`deploy`/`upload` run and die on missing `AWS_ROLE_ARN`/`LIGHTSAIL_SSH_KEY`	gone
`aws/` files	live (deployer image + template)	unchanged and still live

Net: this MR removes the reds that CI itself caused, and leaves exactly one red standing -- the Duo lane -- whose cause is an unassigned entitlement and whose fix is not a CI change.

Pipeline shapes after this MR

Project-defined jobs only; template scanners noted separately. Derived by evaluating each job's own rules: (with extends: resolved), then cross-checked against GitLab's CI Lint API for the two shapes a dry_run can simulate.

Pipeline	Project jobs	Notable
MR from a feature branch touching only CI files (this MR)	17	`build_analyzer_image` + all 14 test-stage jobs (`duo_cli_canary` and the 3 batch jobs included) + `container_scanning` + `compute_version` (dry run). No `build_deployer_image` (its `changes:` list is `aws/**`, `compose.yml`, uploader inputs), no `release`. Plus SAST / secret detection / dependency scanning
MR touching `stig_tools/` only	16, or 17	same as above minus `compute_version`; 17 if the diff touches a deployer-watched file (`stig_tools/stigman_uploader.py`, the sample `.ckl`, the XCCDF), which adds `build_deployer_image`. Plus the same template scanners
MR touching only `CHANGELOG.md`	2	the pipeline still exists (workflow rule 3 admits any `merge_request_event` diff), but runs only `duo_cli_canary` and the `compute_version` dry run -- no build, no tests, no scan. Intentional: a CHANGELOG-only diff changes the computed version and nothing else
Push to `master`	19	the 17 above + `build_deployer_image` + `release` (`when: manual`, `allow_failure: true`). Plus 5 template jobs = 24 total, matching the lint API exactly
Tag pipeline	17	both image builds + all 14 test-stage jobs + `container_scanning`. No release stage -- `compute_version`/`release` are default-branch/MR-only, so a tag pipeline cannot re-release. Matches the lint API's 17 exactly

No dangling needs: in any of the four, checked mechanically. It holds by construction: every build/test job shares one anchor, so a job and everything it needs are always instantiated together; container_scanning's six if: conditions are identical to that anchor's; and release's rule set is a strict subset of compute_version's, so release can never exist without it. Nothing is unreachable -- every job appears in at least one shape, release included.

Verification

glab ci lint .gitlab-ci.yml -> CI/CD YAML is valid! (templates/stig-compliance-analysis.yml is unchanged by this MR and is not standalone-lintable -- it is a component fragment that inherits the consumer's stages:; it fails identically on master.)
CI Lint API (POST /projects/:id/ci/lint, dry_run: true, include_jobs: true) against the final file:
- ref: master -> valid: true, errors: [], warnings: [], 24 jobs, with release reported as when: manual, allow_failure: true.
- ref: v1.0.6 (tag) -> valid: true, errors: [], 17 jobs, no release stage.
- Both counts match the independent rule enumeration above, which is what lets that enumeration be trusted for the two MR shapes dry_run cannot simulate (it only models push).
compute_version executed for real, not reasoned about: extracted from the final .gitlab-ci.yml, run against a fresh full-fetch clone of this repository (all 7 v1.0.x tags present) with this MR's CHANGELOG.md:
```
Highest released: 1.2.4   bump: minor   ->   next tag: v1.3.0
NEW_TAG=v1.3.0
```
Three control runs, same harness: Unreleased with no Added/Changed -> v1.2.5 (patch path); RELEASE_VERSION=2.0.0 -> v2.0.0; RELEASE_VERSION=v2.0.0 -> rejected, exit 1; RELEASE_VERSION=1.0.6 -> duplicate-tag refusal, exit 1.
Shell syntax: bash -n on the extracted compute_version block, sh -n on the extracted release guard. Both clean.
The three-way guard executed in the real release image. Pulled registry.gitlab.com/gitlab-org/cli:latest (Alpine, git 2.54.0 present) and ran the extracted guard against this repository's real remote: NEW_TAG=v1.0.6 -> rc=0, "already exists", job exit 1; NEW_TAG=v1.3.0 -> rc=2, "is free", exit 0; same tag against an unreachable remote -> rc=128, "Refusing to mint blind", exit 1.
Still not verified live: the release: keyword actually minting v1.3.0. It cannot be exercised without creating a real tag, and it is behind the manual gate. Called out in open risks rather than asserted.

What the adversarial review changed

Each of these reverses something in the first draft of this branch.

Finding	What changed
C1 version math wrong (`1.0.7` bare, not `1.2.5`)	Section 6 -- CHANGELOG headings joined the tag scan, `v`-prefix restored, minor/patch rule added, executed and proven `v1.3.0`
C2 Duo presence-gate is a no-op on an unprotected variable, and presence cannot fix rejection	Section 3 -- gate deleted entirely, lane left red on purpose, no "master goes green" claim
C3 `GITLAB_TOKEN: $CI_JOB_TOKEN` is a net regression	Section 8 -- removed; `git ls-remote` guard instead
H1 `release_to_catalog` does not publish (`isCatalogResource: false`)	Section 7 -- renamed to `release`; no catalog claim anywhere; marking the project a resource noted as a separate decision
H2 this MR's own pipeline runs essentially nothing	Section 4 -- `changes:` widened to CI surface; this MR's pipeline now runs the real lane
M1 "add the variable, Protected + Masked" is stale	Required action -- the variable exists; the gap is entitlement. Flipping it to Protected is explicitly not recommended
M2 `aws/lightsail-deploy.sh` is not removable	Section 2 -- kept, with the reason recorded in-file so it is not re-proposed

Kept from the first draft unchanged, per the review's "sound" list: the four AWS anchor removals and the empty deploy stage, the container_scanning repoint with its hand-copied rules, the workflow: rule-ordering fix (so a merge bundling docs with code still produces a master pipeline), and the CONTAINER_SCANNING_DISABLED kill switch.

Required action

Give the Duo lane an entitled identity. GITLAB_DUO_TOKEN already exists as an unprotected, masked + hidden project CI/CD variable holding a project-bot token with api scope. It authenticates. The Duo Agent Platform refuses it (AgenticChatForbiddenError) because the bot holds no Duo entitlement. The fix is a Duo seat for that bot identity, or replacing the value with a token belonging to an entitled user.

Do not flip the variable to Protected. Only master is protected here, so Protected would newly exclude the variable from every MR and feature-branch pipeline -- a behaviour change, not the no-op the first draft implied.

Until then the Duo lane is red on every ref, by design.

None linked. Direct continuation of the CI-health review of this project's own pipeline, redrawn against the absorbed base.

Expect a rebase: !4 (merged) (build/pin-duo-cli, the Duo JSON-contract rework) also touches .gitlab-ci.yml. Whichever lands second rebases. The overlap is narrow -- !4 (merged) works on duo_cli_canary and the stig_tools parsing contract; this MR does not touch either.

Documentation updates

STIGMAN_DEPLOYMENT.md -- rewritten as described in section 2.

No README.md change: its STIG Manager references (STIGMAN_AWS_ROLE_ARN, STIGMAN_LIGHTSAIL_SSH_KEY, STIGMAN_ADMIN_PASSWORD) are the component's consumer-facing inputs, not this project's removed lane, and are accurate as written.

CHANGELOG.md updated

Yes, under ## [Unreleased]: Removed (AWS lane, with the aws/-stays correction), Changed (STIGMAN_DEPLOYMENT.md), and three Fixed entries (MR pipelines ran nothing; the release lane's version/naming/gating/guard; the Duo lane's real cause).

Risks / follow-ups

The mint itself is unexercised. The guard is verified in the real image (above), but nothing verifies the release: keyword creating v1.3.0 short of clicking the manual job. The manual gate is the mitigation: a human is present for the first mint and can read the log.
release does not gate on the test stage. By design, so the Duo entitlement cannot block shipping -- but it means a human could click through a red test lane. The gate is judgement, not enforcement.
compute_version now depends on CHANGELOG.md formatting. A released heading that stops matching ## [X.Y.Z], or an ### Added heading omitted from a feature's Unreleased entry, changes the computed version. The rule is commented in-file and the MR dry run surfaces the result before merge.
container_scanning's six if: conditions are still hand-copied from .full_pipeline_rules. The changes: lists share an anchor now, so only the conditions can drift.
Wider changes: costs more CI minutes. Every MR touching .gitlab-ci.yml, templates/ or scripts/ now builds an image and runs the test lane. That is the point, but it is a real cost increase.
Catalog positioning is still open. This project is not a CI/CD Catalog resource. Marking it one would change what release should do and is a separate decision.
Rebase against !4 (merged) is expected, as above.

Checklist

README.md updated (if applicable) -- not applicable, see Documentation updates
CHANGELOG.md updated
Security documentation updated (if applicable) -- not applicable
Usage examples added (for new features) -- not applicable; CI health fix
Pipeline passes -- it will not, and that is expected. This MR's own pipeline is red at stig::verify-creds for the entitlement reason in section 3. Everything else in it -- build, non-Duo tests, duo_cli_canary, container scanning against the real image, the compute_version dry run -- should be green, and that set is the actual review evidence. A green stig::verify-creds is not achievable from this MR.

ci: ephemeral-first pipelines and a human-gated release