Outbound connections from Gitter environment sometimes fail.
I've noticed that outbound connections from EC2 instances within gitter environment randomly hang. I first noticed the problem during provisioning new CI instances: https://gitlab.com/gitlab-com/gl-infra/gitter-infrastructure/merge_requests/105
With the help of @andrewn we found that running
for i in $(seq 1 20); do curl -vi api.ipify.org; done
on any instance can reliably simulate the problem. We've tested it on
-
mongo-replica-01.prod.gitter
access through bastion https://gitlab.com/gitlab-com/gl-infra/gitter-infrastructure#ssh-to-boxes -
jenkins-slave-01.beta.gitter
(pem key in gitter 1password vault) -
ci-runner-01.prod.gitter
(pem key in gitter 1password vault)
graph LR;
A[production-vpc]--production-internal-network-acl-->B[production-private-subnet-1d];
B--security group-->C[jenkins-slave-01]
We've looked at the security groups and ACL settings and didn't notice anything strange.
@andrewn gave me a recommendation to ask for an opinion/help from @craig and @pharrison? Could you please let me know what approaches/processes would you recommend for pinpointing this issue?
Todo
-
Turn off flow logs connected to gitter-vpc-flowlogs
S3 bucket
Edited by Eric Eastwood