Gitter SSH proxy bastion fails to connect to many boxes when running Ansible command

Gitter SSH proxy bastion fails to connect to many boxes when running Ansible command. It especially happens with the prod inventory that has lots of boxes to connect to.

Example Ansible command where we add SSH keys: https://gitlab.com/gitlab-com/gl-infra/gitter-infrastructure#ssh-to-boxes

Example: https://gitlab.com/gitlab-com/gl-infra/infrastructure/issues/5496

Another example when I was trying to add SSH keys (look for unreachable=1),

PLAY RECAP **************************************************************************************************************************
apps-01.prod.gitter        : ok=4    changed=1    unreachable=0    failed=0
apps-02.prod.gitter        : ok=2    changed=1    unreachable=1    failed=0
bastion-01.prod.gitter     : ok=4    changed=1    unreachable=0    failed=0
cube-01.prod.gitter        : ok=4    changed=1    unreachable=0    failed=0
cube-02.prod.gitter        : ok=4    changed=1    unreachable=0    failed=0
cube-arbiter.prod.gitter   : ok=4    changed=1    unreachable=0    failed=0
es-01.prod.gitter          : ok=1    changed=0    unreachable=1    failed=0
es-02.prod.gitter          : ok=4    changed=1    unreachable=0    failed=0
logging-01.prod.gitter     : ok=2    changed=1    unreachable=1    failed=0
logging-02.prod.gitter     : ok=3    changed=1    unreachable=1    failed=0
master-01.prod.gitter      : ok=0    changed=0    unreachable=1    failed=0
master-02.prod.gitter      : ok=0    changed=0    unreachable=1    failed=0
master-03.prod.gitter      : ok=0    changed=0    unreachable=1    failed=0
mongo-replica-01.prod.gitter : ok=4    changed=1    unreachable=0    failed=0
mongo-replica-02.prod.gitter : ok=4    changed=1    unreachable=0    failed=0
mongo-replica-03.prod.gitter : ok=2    changed=1    unreachable=1    failed=0
mongo-replica-arbiter.prod.gitter : ok=4    changed=1    unreachable=0    failed=0
neo4j-001.prod.gitter      : ok=4    changed=1    unreachable=0    failed=0
redis-01.prod.gitter       : ok=4    changed=1    unreachable=0    failed=0
redis-02.prod.gitter       : ok=3    changed=1    unreachable=1    failed=0
sentinel-01.prod.gitter    : ok=4    changed=1    unreachable=0    failed=0
sentinel-02.prod.gitter    : ok=1    changed=0    unreachable=1    failed=0
sentinel-03.prod.gitter    : ok=1    changed=0    unreachable=1    failed=0
typeahead-01.prod.gitter   : ok=4    changed=1    unreachable=0    failed=0
typeahead-02.prod.gitter   : ok=4    changed=1    unreachable=0    failed=0
webapp-01.prod.gitter      : ok=3    changed=1    unreachable=1    failed=0
webapp-02.prod.gitter      : ok=4    changed=1    unreachable=0    failed=0
webapp-03.prod.gitter      : ok=4    changed=1    unreachable=0    failed=0
webapp-04.prod.gitter      : ok=1    changed=0    unreachable=1    failed=0
webapp-05.prod.gitter      : ok=4    changed=1    unreachable=0    failed=0
webapp-06.prod.gitter      : ok=4    changed=1    unreachable=0    failed=0
webapp-07.prod.gitter      : ok=4    changed=1    unreachable=0    failed=0
webapp-08.prod.gitter      : ok=3    changed=1    unreachable=1    failed=0
ws-01.prod.gitter          : ok=4    changed=1    unreachable=0    failed=0
ws-02.prod.gitter          : ok=4    changed=1    unreachable=0    failed=0
ws-03.prod.gitter          : ok=4    changed=1    unreachable=0    failed=0
ws-04.prod.gitter          : ok=4    changed=1    unreachable=0    failed=0
ws-05.prod.gitter          : ok=3    changed=1    unreachable=1    failed=0
ws-06.prod.gitter          : ok=4    changed=1    unreachable=0    failed=0
ws-07.prod.gitter          : ok=4    changed=1    unreachable=0    failed=0
ws-08.prod.gitter          : ok=4    changed=1    unreachable=0    failed=0

cc @andrewn

Edited by Eric Eastwood