chore: absorb upstream 1.2.4 (fork parent)

Andrew Dunnrequested to merge
absorb/upstream-1.2.4 into master

What does this MR do?

Merges 63 commits from the fork parent (gl-demo-ultimate-britchie2/uncle-sam/stig-analyzer, Bryan Ritchie's project) into this project's master, via a single merge commit rather than a series of cherry-picks. The fork point is f767e70, which is exactly the fork parent's tag v1.0.7; the absorbed range covers releases 1.2.0 through 1.2.4, dated 2026-06-16 through 2026-06-25, all from the same upstream author.

Why absorb everything in one merge, not cherry-pick: a scratch merge of upstream/master onto this project's current master tip resolved with zero conflicts across all 63 commits. One commit in that range renames most of the job names across .gitlab-ci.yml and templates/stig-compliance-analysis.yml (the stig::/stig::stigmanager:: prefixing, below); cherry-picking individual fixes and features out of order would touch those same lines repeatedly and manufacture conflicts that the full merge does not have. This project's only post-fork commit -- Renovate enrollment (renovate.json, !1 (merged)) -- is untouched by the merge and confirmed intact afterward.

Net change: 20 files, +3564/-711.

The delta, by category

Features

  • stig::plan (stig_tools/remediation_planner.py plus a new stateless parser, stig_tools/ckl_parser.py) -- turns an analysis .ckl (Checklist) artifact into a GitLab issue hierarchy: one parent "STIG Remediation Plan" issue plus one child issue per Open finding, with Category (CAT I/II/III) and severity labels, code locations, and a provenance trail back to the run that produced it. Idempotent across re-runs.
  • stig::remediate (stig_tools/remediate.py, new) -- for each open finding, builds a fix prompt from the finding's fix text and code locations, invokes GitLab Duo to edit the code, and opens a merge request only if Duo produced a real diff. Supports an auto-discover mode (fix every open finding) and chaining after stig::plan in one pipeline run. Never auto-merges.

Consumer-driven fixes (each traced in the absorbed commits to a real external consumer hitting the issue):

  • Batch-job generator now caps aggregate needs: at 50, the actual GitLab ceiling on how many jobs a single job may need: -- stricter than the 200-job parallel-matrix cap the code previously enforced.
  • Tag resolution for the CI/CD Catalog now accepts non-v-prefixed tags (see the tag-format note below).
  • New deploy_stage input so a consumer without a deploy stage can point the STIG Manager jobs at a stage it actually has.
  • GIT_STRATEGY/GIT_DEPTH tuning so jobs that need nothing from the consumer's repository skip the clone, and jobs that only read source shallow-clone.
  • Up-front SSH key validation (ssh-keygen -y -f), scoped to templates/stig-compliance-analysis.yml (the reusable component's provision/deploy jobs) -- this project's own AWS scripts are unchanged by the absorb. aws/lightsail-deploy.sh gains SSH failure diagnostics that classify the real cause (key rejected vs. unreachable) instead of reporting every failure as a generic timeout, but one honest gap in that new diagnostic: it tells the operator to check STIGMAN_LIGHTSAIL_SSH_KEY (the template's CI/CD variable name), while the script itself reads and guards on LIGHTSAIL_SSH_KEY -- the two names refer to the same key by different paths through the template/script boundary, and the message names the wrong one.

Duo CLI path change -- read this before relying on it

Upstream dropped the npm @gitlab/duo-cli package entirely. The analyzer image (stig_tools/Dockerfile) no longer runs npm install --global @gitlab/duo-cli; instead it installs glab (GitLab's official command-line tool) from a pinned .deb package, then pre-downloads the Duo CLI's compiled binary (pinned version, SHA-256-verified) into the image at both /usr/local/bin/duo and glab's expected binary directory, and sets glab config set --global duo_cli_auto_run/auto_download true so CI never blocks on an interactive prompt. stig_ai_analyzer.py's call_gitlab_duo() now shells out to glab duo cli run --goal <prompt> instead of invoking duo run directly. stig_tools/verify_duo_creds.sh was rewritten to match: it probes glab api user and distinguishes a live token (JSON with a username field) from a revoked one (invalid_token/Token was revoked), rather than the old duo run-based probe.

Upstream's stated reason for the change (their CHANGELOG: the npm package was deprecated in GitLab 19.1 and duo run fails the instance-version auth handshake in CI) is dated June 2026. A live check run on this date against gitlab.com could not reproduce it: npm @gitlab/duo-cli@9.10.0 authenticates successfully in both text and JSON output modes, the same glab duo cli run invocation shape authenticates identically when run with a current glab (the absorbed pin combination never reaches authentication -- see Known limitations below), the npm package carries no deprecation notice, and the one failure seen on either path traced to an unrelated namespace entitlement gate, not authentication. The absorbed code comments (stig_tools/Dockerfile lines 16-20 and the verify_duo_creds.sh header) state the June diagnosis as settled fact; that will need correcting in the follow-up below, not repeated as-is.

Known limitations of the absorbed Duo CLI path (found running it in the built container, not from reading the diff)

  • The pinned glab version (1.105.0) cannot run any Duo CLI today: it supports Duo CLI major version 8 only, but resolves the package registry's latest release (9.10.0, major 9), and refuses to run it.
  • The image's pre-downloaded binary (pinned 8.107.0) is never reached by that failure -- the run fails identically whether the pre-downloaded binary is present or removed.
  • Setting duo_cli_auto_download=false does not avoid the failure; it instead crashes on an interactive download-confirmation prompt with no TTY attached (the CI environment this is meant to run in).
  • The Dockerfile's own build-time probe of this path (glab duo cli version || echo "note: ...") is ||-swallowed, so the image builds and reports green with the mechanism already dead at build time. Left as absorbed, every analyzer call to Duo would return an error and every STIG rule would silently become Not_Reviewed.
  • A newer glab (1.113.0, tested) resolves the version mismatch above by auto-downloading 9.10.0-modern -- which overwrites the pinned binary with the AVX2-optimized build the Dockerfile's own comment deliberately avoids pinning, for the SIGILL-on-non-AVX2-runners reason that comment states.
  • The pinned Duo CLI binary (8.107.0) predates --output-format json (confirmed first available in 8.110.0 by flag bisection), so even a successful run on this pin could not be asked for structured output.

Planned follow-up: a reconciliation MR restoring a working Duo CLI path -- the compiled binary at a current pin (9.10.0), downloaded and SHA-256-verified directly rather than through glab's version-coupled manager, invoked as duo run rather than glab duo cli run. That change also removes the Node.js base image dependency, since nothing in the image would need npm for a directly-downloaded binary.

Job renames and other changes

  • Jobs renamed under stig:: and stig::stigmanager:: prefixes throughout .gitlab-ci.yml and templates/stig-compliance-analysis.yml (input names are unchanged). .gitlab-ci.yml passes glab ci lint standalone after the merge. templates/stig-compliance-analysis.yml does not: standalone lint now fails with chosen stage stig-evaluate does not exist, because a 1.2.2 commit in this absorb changed the stage input's default from test to stig-evaluate and the template no longer declares that stage itself. It still lints fine in an actual consumer context, where the consumer's own pipeline declares stig-evaluate (the documented single-stage setup this component now expects) -- this is upstream's intent, not a regression this MR introduces, but before this absorb the template was standalone-lintable and after it, it is not.
  • compute_version reworked: it now emits bare (non-v-prefixed) semver tags (e.g. 1.2.1) so a consumer can include the component as @1.2.1 rather than @v1.2.1, and accepts an optional RELEASE_VERSION CI variable to override the automatic patch bump for a minor/major release. It scans both the legacy v* tags and the new bare tags when computing "latest," and the RELEASE_VERSION path explicitly checks both the bare and v-prefixed form of the requested tag before creating it. Caveat for this project specifically: this project's own tags stop at v1.0.6 (v-prefixed); the fetched-and-merged upstream tags (v1.0.7, v1.2.0, 1.2.1-1.2.4) exist in the merged commit history but are upstream's release history, not this project's, and a plain branch push does not carry tags along with it. If those upstream tags are ever pushed to this project's own tag namespace, a subsequent automatic release here would compute its "next" version from 1.2.4 and emit 1.2.5 -- skipping straight past this project's actual last real tag (v1.0.6) and implying four releases (1.1.0 through 1.2.4) that this project never cut. Recommend not pushing the absorbed upstream tags to this project's tag namespace, and treating this project's next real release as one patch past v1.0.6.
  • README restructured -- detail moved out into new docs/FEATURES.md and docs/INPUTS.md, the top-level file shortened.
  • A hard-coded AWS account identifier in STIGMAN_DEPLOYMENT.md replaced with a placeholder.

Verification performed

  • git merge upstream/master onto this project's master: clean, no conflicts.
  • renovate.json (this project's sole post-fork commit) confirmed unchanged after the merge.
  • glab ci lint .gitlab-ci.yml on the merged tree: valid. glab ci lint templates/stig-compliance-analysis.yml standalone: invalid (see the stage-default note above; this is expected in the component's actual usage context).
  • Python test suite: the CI-mirroring command (python3 -m unittest discover -s tests -p "test_ai_*.py" -v, with fakeredis available) passes 11/11. The full tests/ directory (which also covers the new planner, parser, and remediate modules, not yet wired into a dedicated CI job pattern) passes 63/63.
  • git log --oneline origin/master..HEAD | wc -l reports 65: the 63 absorbed commits plus this merge commit. (A second, follow-up commit adding a CHANGELOG.md note, described below, sits on top of that.)
  • The "Known limitations" and auth-recheck findings above come from a separate review pass that built the image and ran the Duo CLI path inside the container, not from reading the diff alone.

Related issues

None filed on this project. The absorbed commits reference the fork parent's own issues and epics (its epic #2/#3, issue #13, #46) for provenance; those do not exist in this project's tracker and are not actionable here. Follow-up work this MR surfaces but does not do: deciding this project's own tagging/release cadence relative to the absorbed version numbers (see the tag-format caveat above); the reconciliation MR for a working Duo CLI path named under "Known limitations" above; and re-evaluating each of !2-!5 against this new base (next section).

Documentation updates

Carried in by the absorbed commits (all upstream's own documentation work, not authored in this MR):

  • README.md restructured; new docs/FEATURES.md and docs/INPUTS.md.
  • STIGMAN_DEPLOYMENT.md: AWS account identifier scrubbed to a placeholder, SSH key-pair guidance expanded.
  • CHANGELOG.md: upstream's own entries for 1.2.4 down to 1.0.7 are now part of this file, plus one [Unreleased] note from a follow-up commit on this branch (see next section).

CHANGELOG.md updated

Yes. The merge brings upstream's own CHANGELOG.md history with it, already in Keep a Changelog format, covering every item in the categories above under dated version headings (1.2.4 down to 1.0.7). On top of that, a second commit on this branch adds an [Unreleased] note stating plainly that those dated headings are the fork parent's release history, not this project's -- this project's own tags stop at v1.0.6. That note matters concretely because release_to_catalog publishes CHANGELOG.md verbatim as the GitLab Release description; without it, the next real release this project cuts from this base would carry release notes that read as if this project had already shipped 1.1.0 through 1.2.4 itself. If this project cuts its own next release from this base, that release's own dated entry is separate follow-up work, informed by the tag-format caveat above.

Impact on open merge requests

Four merge requests are open against this project's master and will need to be rebased and re-evaluated once this lands, before any of them merge:

  • !2 (fix: repoint component images and consumer paths to this project) -- touches stig_tools/generate_batch_pipeline.py, templates/stig-compliance-analysis.yml, CHANGELOG.md, README.md, and STIGMAN_DEPLOYMENT.md (not the Dockerfile). CHANGELOG.md and STIGMAN_DEPLOYMENT.md are both heavily rewritten by this absorb, so conflict risk on rebase concentrates in those two files; the other three are lower-risk but still need a re-check.
  • !3 (feat: extract CCI identifiers from XCCDF parsing) -- touches stig_ai_analyzer.py's parsing path, plus README.md and CHANGELOG.md. Needs a rebase and a re-check against this base, same concentration-of-risk note as !2 for the two shared files.
  • !4 (build: pin @gitlab/duo-cli to an exact version) -- its premise is only partially removed, so this needs reworking, not closing. stig_tools/Dockerfile no longer installs the npm package this MR pins, but app/.gitlab-ci.yml:228 (this project's own demo consumer pipeline) still runs an unpinned npm install --global @gitlab/duo-cli, and !4's fix there is still needed. !4's Renovate custom-regex manager for the Dockerfile's ARG GKG_VERSION also still applies and survives this absorb unchanged. What's new: this absorb adds two more unmanaged pin sites in the same Dockerfile -- ARG GLAB_VERSION=1.105.0 and ARG DUO_CLI_VERSION=8.107.0 (plus its DUO_CLI_SHA256 pin) -- that the Renovate work in !4 should be extended to cover. (Separately, see the "Known limitations" section above: the 1.105.0/8.107.0 pins !4 would extend Renovate coverage to are not currently a working combination.)
  • !5 (ci: add Duo CLI output-contract canary) -- its premise is that stig_ai_analyzer.py invokes duo run directly in text mode; after this absorb that call site invokes glab duo cli run instead. The canary's assumptions about what "the output contract" is need re-evaluation against the new invocation path -- and, per the "Known limitations" section, against the fact that the invocation does not currently succeed at all -- before this MR can land as written.

Merging this MR itself awaits maintainer sign-off; it is not merged as part of drafting this description.

  • README.md updated (if applicable) -- via absorbed upstream commits
  • CHANGELOG.md updated -- absorbed upstream's 1.2.0-1.2.4 entries, plus a follow-up commit's [Unreleased] note distinguishing them from this project's own release history
  • Security documentation updated (if applicable) -- AWS account identifier scrub in STIGMAN_DEPLOYMENT.md, carried in by the absorb
  • Usage examples added (for new features) -- carried in by the absorb (README/docs/FEATURES.md); no new examples authored in this MR
  • Pipeline passes -- not yet run; branch not pushed (draft only, merge frozen pending maintainer sign-off)

Merge request reports