Set sysctl TCP keepalive settings with recommended values for Google Kubernetes Engine
In #2440 (comment 486117933), I discovered mailroom was using the default TCP keepalive settings:
$ sysctl -a | grep tcp_keep
net.ipv4.tcp_keepalive_intvl = 75
net.ipv4.tcp_keepalive_probes = 9
net.ipv4.tcp_keepalive_time = 7200
However, as https://cloud.google.com/compute/docs/troubleshooting/general-tips mentions:
Note that idle connections are tracked for a maximum of 10 minutes, after which their traffic is subject to firewall rules, including the implied deny ingress rule. If your instance initiates or accepts long-lived connections with an external host, you should adjust TCP keep-alive settings on your Compute Engine instances to less than 600 seconds to ensure that connections are refreshed before the timeout occurs.
They recommend the following settings:
sudo /sbin/sysctl -w net.ipv4.tcp_keepalive_time=60 net.ipv4.tcp_keepalive_intvl=60 net.ipv4.tcp_keepalive_probes=5
The Kubernetes issue tracker mentions this: https://github.com/kubernetes/kubernetes/issues/32457#issuecomment-680325785
Since these settings seem to be reasonable defaults, I propose we make these our pod defaults as well.