Log the destinations the airgapped scenario blocks

What does this MR do and why?

The airgapped scenario logs how many packets the OUTPUT policy dropped, but not where they were headed. A destination that none of the four probes names can leave the container, and the job log never shows it. @john.mcdonnell said on #741 that the log-only count "gives visibility without making the job flaky". This MR gives the destinations the same treatment.

  • Right after the allowlist rules, airgap_gitlab_commands appends two rules to the OUTPUT chain. They record the destination of every packet that reaches the policy in a kernel recent list, up to 256 packets per destination.
  • After the specs, report_dropped_destinations logs each destination with its packets, the most first. A count at the limit prints as 256+.
  • Nothing here can fail the job. A rule that cannot be added prints a message in the setup output. A list that cannot be read logs a warning, and so does one whose lines do not parse, with the count and the first such line. The line "none recorded" is kept for an empty list.
flowchart TB
  subgraph run["From setup through the specs, in the GitLab container"]
    direction LR
    PK["Outbound packet"] --> AC["ACCEPT rules"]
    AC -->|"no match"| R2["New rule: record the destination"]
    R2 --> PD["OUTPUT policy DROP"]
  end
  R2 -.->|"adds the address to"| LS[("airgap_dropped list<br/>/proc/net/xt_recent/airgap_dropped")]
  subgraph after["After the specs, in the ensure block of perform"]
    direction LR
    RP["report_dropped_packets<br/>the policy total, as today"]
    RD["New: report_dropped_destinations<br/>one line per destination"]
  end
  LS -.->|"read through docker exec"| RD
  RP --> LOG["Job log lines only: the job result is unchanged"]
  RD --> LOG

Review notes

  • The destination counts matched the policy total in the Docker host run below (869 of 869) and fell 10 short in the CI job (492 of 502). I did not trace the 10. The likely source is packets dropped while the commands between the policy and the tracking rule ran. A destination past the limit, or more than 100 destinations, would also open a gap: the list holds 100 by default and drops the address seen least recently when it is full.
  • The list holds addresses, so it cannot split one destination by port or protocol.
  • Addresses inside the test network show too, because the rules only allow the praefect node, and the existing total already counts that traffic.
  • The first rule only sizes the list. A rule that sets entries cannot carry a hit count, and an unsized list keeps 32 packets per destination (net/netfilter/xt_recent.c in Linux v6.12, lines 357 to 360 and 372 to 377). On the 5.15 kernel of the GitLab-hosted runners these jobs used, a hit count of 1024 was refused and 255 was accepted, so this MR sizes the list at 256.
  • 300 packets sent to one address through these two rules listed 256, and 40 sent to another listed 40.
  • A LOG rule would write to the host's kernel log, and only for containers once the host sets net.netfilter.nf_log_all_netns, which a job cannot count on. The list is read from inside the container instead.

Screenshots or screen recordings

With the rules intact, the scenario ran against gitlab/gitlab-ee:19.4.2-ee.0 on a Docker host, and the 11 smoke examples from five spec files passed. The job output carried the existing lines and the new one:

WARN  -- Airgapped finding: gitlab.com port 8080 is reachable
INFO  -- Airgapped packets dropped by the OUTPUT policy: policy DROP 869 packets, 59248 bytes
INFO  -- Airgapped destinations blocked by the OUTPUT policy, with packets counted (at most 256 per destination)
  172.18.0.3: 168
  172.18.0.4: 166
  172.18.0.5: 166
  52.38.134.47: 108
  184.34.154.174: 102
  184.34.28.19: 89
  172.65.251.78: 21
  104.18.32.27: 14
  104.20.23.154: 14
  172.64.155.229: 14
  8.8.8.8: 7

In Docker-in-Docker 29.9.1 jobs on GitLab-hosted runners the scenario passed the same way, with 502 packets dropped and eleven destinations that account for 492 of them. With tcp port 443 open to every destination the job failed on Airgapped network faulty. 172.65.251.78 port 443 is reachable. in all three attempts and printed no destination line, as before this change. Without the change, the same 11 examples passed and no destination line appeared.

How to set up and validate locally

  1. Run bundle exec rspec spec/gitlab/qa/scenario/test/instance/airgapped_spec.rb.

  2. From this branch, with EE_LICENSE set (and GITLAB_LICENSE_MODE=test for a test license), run the scenario on the image and the five spec files those runs used. Look for the Airgapped destinations blocked line after the dropped packet count.

    S=qa/specs/features
    bundle exec exe/gitlab-qa Test::Instance::Airgapped EE:19.4.2-ee.0 -- --tag smoke \
      $S/api/9_tenant_scale/users_spec.rb \
      $S/api/3_create/repository/add_list_delete_branches_spec.rb \
      $S/api/3_create/merge_request/push_options_spec.rb \
      $S/browser_ui/4_verify/ci_variable/raw_variables_defined_in_yaml_spec.rb \
      $S/api/10_software_supply_chain_security/project_access_token_spec.rb
  3. To see that a leak still fails the job, add iptables -I OUTPUT 1 -p tcp --dport 443 -j ACCEPT after the praefect rule in airgap_gitlab_commands and run step 2 again.

MR acceptance checklist

This checklist encourages us to confirm any changes have been analyzed to reduce risks in quality, performance, reliability, security, and maintainability.

An AI coding assistant wrote this change. The assistant also checked each claim against the Linux v6.12 xt_recent source, airgapped.rb and the CI job logs, and ran RuboCop and the specs on a test VM. It also ran the scenario against GitLab EE 19.4.2 on a test VM and in CI jobs.

Edited by Andrew Dunn

Merge request reports

Loading
Loading