Update avatars nginx to not remember `webapp-0x` DNS lookups (don't point at dead boxes)

Follow-up to https://gitlab.com/gitlab-com/gl-infra/infrastructure/issues/6971

Update avatars nginx to not remember webapp-0x DNS lookups and always point to whatever new server is behind the domain webapp-0x.prod.gitter. I assume this is DNS caching but whatever mechanism it is that tries to connect to the same box.

Restarting nginx seems to kick nginx into seeing the new webapp-0x servers.

This line in the avatars nginx conf that sets up the gitter-avatars-api-backend upstream expands in to the following, /etc/nginx/sites-enabled/gitter-avatars.conf

upstream gitter-avatars-api-backend {
  server webapp-04.prod.gitter:5025 max_fails=3 fail_timeout=10s;
  server webapp-04.prod.gitter:5023 max_fails=3 fail_timeout=10s;
    server webapp-03.prod.gitter:5025 max_fails=3 fail_timeout=10s;
  server webapp-03.prod.gitter:5023 max_fails=3 fail_timeout=10s;
    server webapp-01.prod.gitter:5025 max_fails=3 fail_timeout=10s;
  server webapp-01.prod.gitter:5023 max_fails=3 fail_timeout=10s;
    server webapp-08.prod.gitter:5025 max_fails=3 fail_timeout=10s;
  server webapp-08.prod.gitter:5023 max_fails=3 fail_timeout=10s;
    server webapp-06.prod.gitter:5025 max_fails=3 fail_timeout=10s;
  server webapp-06.prod.gitter:5023 max_fails=3 fail_timeout=10s;
    server webapp-07.prod.gitter:5025 max_fails=3 fail_timeout=10s;
  server webapp-07.prod.gitter:5023 max_fails=3 fail_timeout=10s;
    server webapp-02.prod.gitter:5025 max_fails=3 fail_timeout=10s;
  server webapp-02.prod.gitter:5023 max_fails=3 fail_timeout=10s;
    server webapp-05.prod.gitter:5025 max_fails=3 fail_timeout=10s;
  server webapp-05.prod.gitter:5023 max_fails=3 fail_timeout=10s;
}

This behavior was discovered because we updated the webapp AMI(image) and recycled our whole fleet so it's all new webapp servers. More background in https://gitlab.com/gitlab-com/gl-infra/infrastructure/issues/6971

Edited by Eric Eastwood