Outbound connections from Gitter environment sometimes fail.

I've noticed that outbound connections from EC2 instances within gitter environment randomly hang. I first noticed the problem during provisioning new CI instances: https://gitlab.com/gitlab-com/gl-infra/gitter-infrastructure/merge_requests/105

With the help of @andrewn we found that running

for i in $(seq 1 20); do curl -vi api.ipify.org; done

on any instance can reliably simulate the problem. We've tested it on

• mongo-replica-01.prod.gitter access through bastion https://gitlab.com/gitlab-com/gl-infra/gitter-infrastructure#ssh-to-boxes
• jenkins-slave-01.beta.gitter (pem key in gitter 1password vault)
• ci-runner-01.prod.gitter (pem key in gitter 1password vault)

graph LR;
A[production-vpc]--production-internal-network-acl-->B[production-private-subnet-1d];
B--security group-->C[jenkins-slave-01]

We've looked at the security groups and ACL settings and didn't notice anything strange.

@andrewn gave me a recommendation to ask for an opinion/help from @craig and @pharrison? Could you please let me know what approaches/processes would you recommend for pinpointing this issue?

Todo

• Turn off flow logs connected to gitter-vpc-flowlogs S3 bucket
  • https://gitlab.com/gitlab-com/gl-infra/infrastructure/issues/6656#note_165424858