Use iptables LOG target to preview what would be dropped.

Corrective action from https://gitlab.com/gitlab-com/gl-infra/infrastructure/issues/8528.

Update https://ops.gitlab.net/gitlab-cookbooks/gitlab-iptables

---

Suggested implementation notes:

For a more cautious roll-out, we could initially make any new iptables rules LOG instead of REJECT or DROP.

Unlike most jump targets, LOG is non-terminating; execution falls through to subsequent rules after the LOG action. This would let us preview what packets would be rejected/dropped. After validation, we could optionally keep the LOG rule immediately before the new REJECT or DROP rule if we want to get log messages about unexpected packets.

Caution: Writing a log message per packet could overwhelm logging systems (e.g. write rate to /var/log, Stackdriver collection of syslog, dmesg). To rate-limit the logging, we can optionally append the "limit" module to the LOG rule (but not for the corresponding REJECT/DROP rule).

For reference, the LOG target logs to dmesg and syslog. Its "--log-prefix" message is limited to 29 characters and typically you should end it with whitespace. For more details on the LOG target or the limit module, see the "iptables-extensions" manpage.

Example:

Preview a sample of what packets would be rejected:

$ sudo iptables -A INPUT -p tcp -m tcp --dport 1234 -j LOG --log-level debug --log-prefix 'Would reject TCP 1234 ' -m limit --limit 1/second --limit-burst 10

$ nc -l 1234
$ echo 'hi' | nc localhost 1234

$ dmesg | grep 'Would reject TCP 1234 '
[114477.090576] Would reject TCP 1234 IN=lo OUT= MAC=00:00:00:00:00:00:00:00:00:00:00:00:08:00 SRC=127.0.0.1 DST=127.0.0.1 LEN=60 TOS=0x00 PREC=0x00 TTL=64 ID=48831 DF PROTO=TCP SPT=37672 DPT=1234 WINDOW=65495 RES=0x00 SYN URGP=0 
[114477.090716] Would reject TCP 1234 IN=lo OUT= MAC=00:00:00:00:00:00:00:00:00:00:00:00:08:00 SRC=127.0.0.1 DST=127.0.0.1 LEN=52 TOS=0x00 PREC=0x00 TTL=64 ID=48832 DF PROTO=TCP SPT=37672 DPT=1234 WINDOW=512 RES=0x00 ACK URGP=0 
[114477.090955] Would reject TCP 1234 IN=lo OUT= MAC=00:00:00:00:00:00:00:00:00:00:00:00:08:00 SRC=127.0.0.1 DST=127.0.0.1 LEN=55 TOS=0x00 PREC=0x00 TTL=64 ID=48833 DF PROTO=TCP SPT=37672 DPT=1234 WINDOW=512 RES=0x00 ACK PSH URGP=0 
[114491.964075] Would reject TCP 1234 IN=lo OUT= MAC=00:00:00:00:00:00:00:00:00:00:00:00:08:00 SRC=127.0.0.1 DST=127.0.0.1 LEN=52 TOS=0x00 PREC=0x00 TTL=64 ID=48834 DF PROTO=TCP SPT=37672 DPT=1234 WINDOW=512 RES=0x00 ACK FIN URGP=0 

$ sudo iptables -D INPUT -p tcp -m tcp --dport 1234 -j LOG --log-level debug --log-prefix 'Would reject TCP 1234 ' -m limit --limit 1/second --limit-burst 10

Later apply the REJECT rule, but keep the preceding LOG rule:

$ sudo iptables -A INPUT -p tcp -m tcp --dport 1234 -j LOG --log-level debug --log-prefix 'Reject TCP 1234 ' -m limit --limit 1/second --limit-burst 10
$ sudo iptables -A INPUT -p tcp -m tcp --dport 1234 -j REJECT

$ nc -l 1234
$ echo 'hi' | nc localhost 1234

$ dmesg | grep 'Reject TCP 1234 '
[115303.889010] Reject TCP 1234 IN=lo OUT= MAC=00:00:00:00:00:00:00:00:00:00:00:00:08:00 SRC=127.0.0.1 DST=127.0.0.1 LEN=60 TOS=0x00 PREC=0x00 TTL=64 ID=1199 DF PROTO=TCP SPT=37780 DPT=1234 WINDOW=65495 RES=0x00 SYN URGP=0 
[115304.901900] Reject TCP 1234 IN=lo OUT= MAC=00:00:00:00:00:00:00:00:00:00:00:00:08:00 SRC=127.0.0.1 DST=127.0.0.1 LEN=60 TOS=0x00 PREC=0x00 TTL=64 ID=1200 DF PROTO=TCP SPT=37780 DPT=1234 WINDOW=65495 RES=0x00 SYN URGP=0