Remove residual firewall rule definitions that would re-apply on next reboot

Recently some iptables DROP rules were unintentionally applied to Patroni hosts, causing connectivity loss to the database servers (incident and review). The immediate recovery steps included unassigning Chef role gitlab-iptables and manually remove iptables rules from the affected hosts (especially the Patroni hosts).

That left some residual unmanaged configuration on those hosts. As part of an incident follow-up corrective action, that residual config was discovered and found to probably be dangerous.

The "gitlab-iptables" cookbook's "default" recipe uses cookbook "iptables-ng" to define iptables rules. That "iptables-ng" cookbook installs deb package "iptables-persistent" (and implicitly also the "netfilter-persistent" deb package), whose job is to persist those iptables rules across reboots.

The "iptables-ng" cookbook stores its version of the rules in /etc/iptables.d/[table]/[chain]/[named_rule], but as far as I know, those are only used by chef-client to support that cookbook. So those files by themselves may be harmless.

In contrast, the contents of /etc/iptables/rules.v4 and rules.v6 are actively loaded during reboot by the "netfilter-persistent" systemd service.

For all hosts that unintentionally had the "gitlab-iptables" cookbook applied, if that cookbook has been removed and not reapplied, we should apply some clean up steps that at least include one of the following:

  • Option A: Disable systemd unit "netfilter-persistent.service"
  • Option B: Remove files /etc/iptables/rules.*
  • Option C: Remove deb package "iptables-persistent" and optionally also "netfilter-persistent"