Make the airgapped probes able to fail
What does this MR do and why?
The airgapped scenario's only leak check that can fail is gitlab.com port 80. The wget probe that follows it exits 0 whether or not the fetch succeeds. This MR makes the isolation checks able to fail, as discussed in #741.
- After the rules go in, the job fails if a TCP connect to gitlab.com:443, example.com:80 or :443, or dns.google:53 succeeds.
- gitlab.com:8080 is open in the rules, so
report_gitlab_port_8080only logs whether it is reachable. report_dropped_packetslogs how many packets theOUTPUTpolicy dropped once the specs finish.
flowchart LR
L1["All egress open"] ==> O1["Job fails today and with this change"]
L2["Port 53 open, tcp and udp"] ==> O2["Job fails only with this change"]
L3["Port 443 open"] ==> O2
L4["Port 8080, open in the rules"] ==> O3["Logged by report_gitlab_port_8080, job stays green"]
L5["Port 53 open, udp only"] ==> O4["Job stays green either way"]Review notes
- A failing probe gets two retries before the job fails (
base.rb:213-229). - Each new check waits up to 10 seconds for a connect the rules drop, so a sealed run takes about 40 seconds longer.
- The dropped count includes metrics scrapes of the gitaly nodes on port 9236, which never leave the test network, so the count overstates outbound traffic.
- example.com and dns.google are resolved before the rules go in. example.com:443 must answer then, so a failed lookup or an unreachable example.com fails the job.
- The manual
test-on-omnibusjob runs theairgappedjob from gitlab-org/gitlab on this branch, in the CI network that the example.com and dns.google lookups depend on. I'll play it after this MR opens and link the result before requesting review. - The
prove_airgapcheck in gitlab-org/gitlab has the same shape, and I'll fix it in a separate MR there. Theairgappedjob there picks up this change after a gem release and aqa/Gemfile.lockbump.
Screenshots or screen recordings
When the scenario runs with this change against gitlab/gitlab-ee:19.4.2-ee.0 on a Docker host, with the rules intact, 11 smoke examples from five spec files pass. The job output carries Airgapped finding: gitlab.com port 8080 is reachable at warn level and Airgapped packets dropped by the OUTPUT policy: policy DROP 1245 packets, 81808 bytes at info level.
With tcp port 443 open to every destination, all three attempts fail on Airgapped network faulty. 172.65.251.78 port 443 is reachable. and the job fails. The port 53 rows in the diagram come from the same rules run in a plain container, not from a scenario run.
Without the change, the same 11 examples pass and neither line appears. In Docker-in-Docker 29.9.1 jobs on GitLab-hosted runners, the scenario gave the same results in all three cases. Only the dropped count differed. In those jobs, example.com, dns.google and gitlab.com resolved before the rules went in.
How to set up and validate locally
-
Run
bundle exec rspec spec/gitlab/qa/scenario/test/instance/airgapped_spec.rb. -
From this branch, with
EE_LICENSEset (andGITLAB_LICENSE_MODE=testfor a test license), run the scenario on the image and the five spec files those runs used. Look for the two log lines above.S=qa/specs/features bundle exec exe/gitlab-qa Test::Instance::Airgapped EE:19.4.2-ee.0 -- --tag smoke \ $S/api/9_tenant_scale/users_spec.rb \ $S/api/3_create/repository/add_list_delete_branches_spec.rb \ $S/api/3_create/merge_request/push_options_spec.rb \ $S/browser_ui/4_verify/ci_variable/raw_variables_defined_in_yaml_spec.rb \ $S/api/10_software_supply_chain_security/project_access_token_spec.rb -
To see a leak fail, add
iptables -I OUTPUT 1 -p tcp --dport 443 -j ACCEPTafter the praefect rule inairgap_gitlab_commands. Run step 2 again. Each attempt fails onport 443 is reachable, and the job fails. A rule added by hand to one container is gone after the retry, which starts a new container.
MR acceptance checklist
- Commits include
Changelog:trailer - I have evaluated the MR acceptance checklist for this MR.
An AI coding assistant wrote this change. It also ran RuboCop and the specs, and ran the scenario against GitLab EE 19.4.2 on a test VM and in CI jobs.