Make the airgap check in the QA runner able to fail

What does this MR do and why?

The airgapped E2E job checks that the runner is isolated after it registers it. That check has never been able to fail: it passes whether or not the runner reaches the internet. This MR makes it fail the job when a probe connects, which is the follow-up promised in gitlab-qa!1477 (merged) for the problem described in gitlab-qa#741.

  • Each probe was written as probe && (exit 1) || (exit 0). When the probe connects, (exit 1) ends only its own subshell, so the || branch runs and the check exits 0. Each probe is now an if, and one that connects exits 1.
  • The wget probe passed GNU-only flags such as --retry-connrefused. The default gitlab-runner:alpine image has busybox wget, which rejects them with unrecognized option, so the probe never sent a packet. It now uses -q -T 10 -O /dev/null, which busybox and GNU wget both accept.
flowchart TD
  A["Runner container is up"] --> B{"Does registry.gitlab.com resolve where the tests run?"}
  B -- no --> C["Nothing is probed, job continues"]
  B -- yes --> D{"Does netcat connect to port 80?"}
  D -- yes --> E["Job fails"]
  D -- no --> F{"Does wget fetch the registry page?"}
  F -- yes --> E
  F -- no --> G["Job continues"]

Before this change both "yes" branches also ended in "Job continues".

Review notes

  • Only a probe that connects fails the job. A timeout, a refused connection or a failed lookup passes, as it did before, so a flaky resolver cannot fail the job. @john.mcdonnell said leak checks that can actually fail would be a clear improvement in gitlab-qa#741 (comment 4019937419), and this keeps the job from failing for any other reason.
  • The airgapped scenario creates the runner network with --internal (https://gitlab.com/gitlab-org/gitlab-qa/-/blob/5048bbbca1b33055b4719e5cc3afe3d9dd6f0107/lib/gitlab/qa/component/base.rb#L99-109), where both probes fail with Network unreachable or bad address.
  • The wget probe now runs, so a check on an internal network takes about 5 seconds longer, spent on the name lookup. On a network that drops packets without answering, it takes 20 seconds instead of 10.
  • The early return when registry.gitlab.com does not resolve is unchanged.
  • qa/Gemfile.lock is untouched. The gitlab-qa probes from the MR above will reach this project's airgapped job separately, after a gem release.

Screenshots or screen recordings

Output of the command prove_airgap builds, run with docker exec in registry.gitlab.com/gitlab-org/gitlab-runner:alpine on Docker 29.9.3:

Runner network Before After
--internal, as the scenario creates it passes passes
Packets dropped without an answer passes after 10 seconds passes after 20 seconds
Internet access passes, with Airgapped network faulty in the output fails with Airgapped network faulty. Connectivity netcat check failed.

With netcat pointed at an unroutable address, so that only wget can see the leak, a network with internet access passes with the old wget flags and fails with the new ones.

How to set up and validate locally

  1. Run cd qa && bundle exec rspec --options .rspec_internal spec/service/docker_run/gitlab_runner_spec.rb. The new #prove_airgap examples run the generated command with stub nc and wget programs. The wget stub rejects long options, as busybox does for --retry-connrefused.
  2. Replace qa/qa/service/docker_run/gitlab_runner.rb with the version on master and run step 1 again. The examples for a connecting netcat and a fetching wget fail.

An AI coding assistant wrote this change, ran the network arms above on a test VM, and ran the specs and RuboCop.

MR acceptance checklist

Evaluate this MR against the MR acceptance checklist. It helps you analyze changes to reduce risks in quality, performance, reliability, security, and maintainability.

Merge request reports

Loading
Loading