Gate the airgap toolchain on base and make its control falsifiable

What this fixes

Two defects in the airgap toolchain, and the second is the one that matters.

The commands were ungated. cmd_airgap_verify drove k3s eight times with no base check and cmd_airgap_mirror assumed k3s as well, so on omnibus the base-agnostic negative control passed while every mirror step behind it no-opped or died. A run confirmed the reassuring half and skipped the proving half, which reads as partial confirmation. That is the worst available failure shape, and one real seal run was scored that way.

The negative control could not fail in one direction. The old check was curl registry.gitlab.com with any non-zero exit read as proof of isolation. curl exits non-zero for a dead resolver (6) as readily as for a blocked route (7, 28), and exits zero on a 404, so a host with egress wide open and only its resolver broken reported a confirmed airgap.

What changed

airgap-mirror and airgap-verify refuse on a base they do not fit, including an unknown base. Refusing on unknown is deliberate: assuming a base is how this got in.

omnibus gets its own airgap path. It installs GitLab as an RPM, so container images are not on its critical path and an image re-pull proves nothing about it. Its mirror is a dnf repository served over loopback, and its positive control is the same tool doing the same operation with only the source changed: dnf against the remote repositories must fail while an install from the mirror succeeds. The mirror step leaves the upstream repositories enabled on purpose, because a makecache that fails for want of any configured remote looks identical to one that fails because the seal held.

The negative control is now a comparison. airgap-seal records whether egress was reachable before it creates any rule and refuses to seal when it was not; airgap-verify refuses a verdict without that baseline and reports INCONCLUSIVE where it cannot tell a seal from a broken probe. New gce/airgap-probe.sh classifies exit codes instead of collapsing them, counts a TLS error as reached because the connection was made, probes IP literals and a high port and a second network, and treats DNS as an observation rather than a verdict.

Evidence

Run on a real omnibus VM, GitLab 19.3.1-ee, under a GCE egress-deny firewall.

• Sealed: five legs blocked, four of them needing no DNS, exit codes 28 and 124. Open: five legs reached. Same probe, both polarities, same host.
• DNS kept resolving under the seal, through the metadata address the allow rule permits. That is exactly the condition that makes a DNS-based control lie, which is why the probe reports it and does not vote on it.
• omnibus positive control passed: nine services up, /users/sign_in rendering 200, remote repositories unreachable, the marker package installed from the mirror, and gitlab-ctl reconfigure completing with no egress.
• Every refusal fired: dev base, unknown base, absent baseline, and a second seal over an existing one.

Three things the run found that reading did not

A GCE seal is not in effect when gcloud returns. Rules created, instance correctly tagged, and the first probe still reached the internet on all five legs; a later one was fully blocked. airgap-seal now polls until the boundary is observable and fails if it never is (AIRGAP_SEAL_WAIT, default 180s). The window produced a false negative, so it was the safe direction, but leaving it teaches that the control is flaky and that re-running until it agrees is normal. That habit is how a real NOT_SEALED gets waved through.

dnf makecache alone does not touch the network. It refetches only metadata it considers expired, so minutes after the mirror step primed the cache it exited 0 having contacted nothing and reported the remote repositories reachable on a sealed host. The leg was measuring cache freshness while claiming to measure reachability. It now runs clean metadata and makecache --refresh.

The mirror served on 8080, which is Puma's. simulation.sh already documented avoiding GitLab's ports and I did not read it. The collision surfaced as "mirror is not answering", and had GitLab answered the mirror's health path the positive control would have been measuring the wrong process. The mirror step now refuses a busy port and names what holds it.

All three are one pattern: a check can pass or fail for a reason other than the one it claims.

What it costs

airgap-seal is slower, because it now runs a baseline probe and then waits for the boundary to become observable. The lab run took 68 seconds end to end. That buys a seal that is confirmed rather than assumed.

airgap-verify on omnibus runs gitlab-ctl reconfigure by default, which is several minutes. It is the operation an offline administrator actually performs and the one most likely to reach for something it cannot have, so it is the strongest proof available here. AIRGAP_VERIFY_SKIP_RECONFIGURE=1 skips it and makes the output say what was therefore not proven.

AIRGAP_ALLOW_UNPROVEN_BASELINE=1 lets you seal a host whose egress was never demonstrated. It does not buy a pass: it records the baseline as unusable so the verify reports INCONCLUSIVE. An override that bought a pass would be the original defect with a flag on it.

Testing

just ci is green, and it now includes just probe-selftest: 15 offline assertions over the probe's exit-code classification and verdict rules, no network and no VM. I reintroduced the exact defect this replaces, watched the self-test fail, and restored it, because a passing test I have never seen fail is not evidence.

The live behaviour above was verified by hand on a throwaway VM, which was torn down. No orphaned instances or firewall rules remain.

Also in here

Two silent-pass bugs fixed in passing. caproni's mirror-pull check ran a 20-iteration wait and then fell through printing nothing, so a mirror serving nothing produced no output and no failure. two-plane verify carried its own inline copy of the weak negative control and now shares the corrected one.

gce/gce.md is corrected on a separate point: it claimed full-stack specs do not run on the dev base. They do, with three fixes that are currently applied by hand and not yet in the bootstrap, plus a Gitaly runtime-directory trap worth knowing about. Folding those into bootstrap-gitlab-dev.sh is left as owed work rather than smuggled into this MR.