windows-ec2-runner: on hold — wiring lands, automated bake deferred

Roger Howardrequested to merge
windows-ec2-runner into main

Status: ON HOLD

The full EC2-orchestrator wiring is here — Terraform AWS layer, Packer config, the Bash orchestrator, the templated PowerShell build script, the CI job, and the docs — but the AMI-bake side hit a string of avoidable-but-stubborn failures (OpenSSH-on-Windows session-close hangs, an AWS snapshot that got stuck at 0% for 14 hours). Rather than burn more EC2 time grinding it out, parking this and returning to it later.

What's parked vs what's live:

Surface State
Local Windows build (MSYS2 + WiX) ✅ works, documented in docs/deployment.md
Tagged-release Linux binaries + Docker images ✅ unchanged from main
Tagged-release Windows zip / MSI ⏸️ not produced — CI path on hold
build-windows-amd64 / build-windows-msi CI jobs ⏸️ gated to RUN_WINDOWS=1 manual triggers
AWS Terraform-managed infra ✅ applied + dormant at $0/mo cost (no instances, no AMI)
Orchestrator + Packer config ✅ on disk, retains all six lessons learned

What's in this MR

Same scope as before — see commit list — but with the CI jobs now gated behind RUN_WINDOWS=1 so they don't fire on every push (the most recent commit on the branch). The wiring stays in the file so a future revisit can resume from this point.

Windows EC2 runner (the original goal)

Static AWS layer (Terraform, deploy/aws/):

  • Dedicated VPC 10.42.0.0/16 + /24 public subnet + IGW
  • Locked-down SG iiiris-windows-build (no inbound — production instances are SSM-only)
  • IAM user iiiris-ci with tag-based ABAC
  • IAM role + instance profile providing SSM Managed Instance Core + S3
  • S3 bucket iiiris-ci-artifacts-<hex> with 7-day lifecycle + SSE

Build AMI (Packer, deploy/packer/):

  • Source: latest Windows_Server-2022-English-Full-Base-*
  • Single combined install-toolchain.ps1 (chocolatey → MSYS2 UCRT64 + libvips + Go 1.25 + WiX v5.0.2 + extensions + git + AWS CLI)
  • cleanup.ps1 resets EC2Launch v2's .run-once marker
  • Communicator: SSH (OpenSSH Server installed via user_data) + PowerShell-as-default-shell

EC2 orchestrator (scripts/aws/ec2-build-windows.sh):

  • Resolves latest AMI by name+tag
  • Launches m5.large, waits for SSM agent
  • Sends AWS-RunRemoteScript pointing at S3-hosted build.ps1 (HTTPS S3 URL — s3:// URIs are rejected by the SSM plugin)
  • Polls, downloads artifacts, terminates
  • trap cleanup EXIT INT TERM for safe teardown on any exit path

Brand + docs (drive-by)

  • logos/02-*.svg + 512px PNG
  • README + all 16 docs open with the wordmark
  • iiiris server: GET / minimal landing page; /logo.svg + /favicon.svg public routes
  • Admin UI sidebar gains the icon mark
  • iiiris-org/gitlab-profile rewritten as a non-technical summary

Journey (lessons captured in commits + code comments)

  1. WinRM hang post-MSYS2 — collapsed three install scripts into one (install-toolchain.ps1).
  2. WinRM never connected — pivoted to SSH (OpenSSH Server + PowerShell-as-default-shell).
  3. ED25519 keypair rejected — Windows AMIs need RSA.
  4. SSH TCP-timeout despite valid SG — switched temp SG to 0.0.0.0/0 since checkip.amazonaws.com returned a wrong egress IP under our caller's CGNAT.
  5. OpenSSH session-close hang (mid-bake) — local DNS hiccup severed SSH 95% through install.
  6. OpenSSH session-close hang (post-script) — [Environment] ::Exit(0) doesn't unblock Packer. Worked around by snapshotting the AMI directly via aws ec2 create-image --no-reboot. But the resulting snapshot then stuck at 0% for 14 hours, broke silently, and one more round of git+aws CLI bake-in work identified more missing tools.

Test plan (when this resumes)

  • Bake a fresh AMI (Packer with current install-toolchain.ps1 now includes git + awscli)
  • Trigger build-windows-amd64 manually with RUN_WINDOWS=1
  • Verify orchestrator launches instance → SSM build → artifact upload → instance terminated cleanly
  • Tag a release with RUN_WINDOWS=1 to validate build-windows-msi

Decision needed before merging

This MR introduces functionality that is gated off. Reasonable paths:

  • Merge anyway — the wiring is reviewable and the CI rules make it dormant. Lets future-us pick up by flipping a variable.
  • Hold the MR open — until the AMI bake works end-to-end.

🤖 Generated with Claude Code

Edited by Roger Howard

Merge request reports