Make the airgapped probes able to fail

What does this MR do and why?

The airgapped scenario's only leak check that can fail is gitlab.com port 80. The wget probe that follows it exits 0 whether or not the fetch succeeds. This MR makes the isolation checks able to fail, as discussed in #741.

  • After the rules go in, the job fails if a TCP connect to gitlab.com:443, example.com:80 or :443, or dns.google:53 succeeds.
  • gitlab.com:8080 is open in the rules, so report_gitlab_port_8080 only logs whether it is reachable.
  • report_dropped_packets logs how many packets the OUTPUT policy dropped once the specs finish.
flowchart LR
  L1["All egress open"] ==> O1["Job fails today and with this change"]
  L2["Port 53 open, tcp and udp"] ==> O2["Job fails only with this change"]
  L3["Port 443 open"] ==> O2
  L4["Port 8080, open in the rules"] ==> O3["Logged by report_gitlab_port_8080, job stays green"]
  L5["Port 53 open, udp only"] ==> O4["Job stays green either way"]

Review notes

  • A failing probe gets two retries before the job fails (base.rb:213-229).
  • Each new check waits up to 10 seconds for a connect the rules drop, so a sealed run takes about 40 seconds longer.
  • The dropped count includes metrics scrapes of the gitaly nodes on port 9236, which never leave the test network, so the count overstates outbound traffic.
  • example.com and dns.google are resolved before the rules go in. example.com:443 must answer then, so a failed lookup or an unreachable example.com fails the job.
  • The manual test-on-omnibus job runs the airgapped job from gitlab-org/gitlab on this branch, in the CI network that the example.com and dns.google lookups depend on. I'll play it after this MR opens and link the result before requesting review.
  • The prove_airgap check in gitlab-org/gitlab has the same shape, and I'll fix it in a separate MR there. The airgapped job there picks up this change after a gem release and a qa/Gemfile.lock bump.

Screenshots or screen recordings

When the scenario runs with this change against gitlab/gitlab-ee:19.4.2-ee.0 on a Docker host, with the rules intact, 11 smoke examples from five spec files pass. The job output carries Airgapped finding: gitlab.com port 8080 is reachable at warn level and Airgapped packets dropped by the OUTPUT policy: policy DROP 1245 packets, 81808 bytes at info level.

With tcp port 443 open to every destination, all three attempts fail on Airgapped network faulty. 172.65.251.78 port 443 is reachable. and the job fails. The port 53 rows in the diagram come from the same rules run in a plain container, not from a scenario run.

Without the change, the same 11 examples pass and neither line appears. In Docker-in-Docker 29.9.1 jobs on GitLab-hosted runners, the scenario gave the same results in all three cases. Only the dropped count differed. In those jobs, example.com, dns.google and gitlab.com resolved before the rules went in.

How to set up and validate locally

  1. Run bundle exec rspec spec/gitlab/qa/scenario/test/instance/airgapped_spec.rb.

  2. From this branch, with EE_LICENSE set (and GITLAB_LICENSE_MODE=test for a test license), run the scenario on the image and the five spec files those runs used. Look for the two log lines above.

    S=qa/specs/features
    bundle exec exe/gitlab-qa Test::Instance::Airgapped EE:19.4.2-ee.0 -- --tag smoke \
      $S/api/9_tenant_scale/users_spec.rb \
      $S/api/3_create/repository/add_list_delete_branches_spec.rb \
      $S/api/3_create/merge_request/push_options_spec.rb \
      $S/browser_ui/4_verify/ci_variable/raw_variables_defined_in_yaml_spec.rb \
      $S/api/10_software_supply_chain_security/project_access_token_spec.rb
  3. To see a leak fail, add iptables -I OUTPUT 1 -p tcp --dport 443 -j ACCEPT after the praefect rule in airgap_gitlab_commands. Run step 2 again. Each attempt fails on port 443 is reachable, and the job fails. A rule added by hand to one container is gone after the retry, which starts a new container.

MR acceptance checklist

An AI coding assistant wrote this change. It also ran RuboCop and the specs, and ran the scenario against GitLab EE 19.4.2 on a test VM and in CI jobs.

Edited by Tiffany Rea

Merge request reports

Loading
Loading