Recent uptick in 504 GATEWAY_TIMEOUT on staging/next - Unable to communicate with Gitter

Over the past week, we have seen a lot of 504 GATEWAY_TIMEOUT on staging/next. I only see 2 events from the production frontend vs 2,000 events on frontend-staging

Unhandled rejection Error: GATEWAY_TIMEOUT
    at h (https://cdn02.gitter.im/_s/280f48a69/js/default.chunk.js:1:10826)
    at Object.error (https://cdn02.gitter.im/_s/280f48a69/js/default.chunk.js:1:12299)
    at d (https://cdn03.gitter.im/_s/280f48a69/js/vendor.chunk.js:1:137766)
    at Object.fireWith [as rejectWith] (https://cdn03.gitter.im/_s/280f48a69/js/vendor.chunk.js:1:138573)
    at l (https://cdn03.gitter.im/_s/280f48a69/js/vendor.chunk.js:1:185299)
    at XMLHttpRequest.<anonymous> (https://cdn03.gitter.im/_s/280f48a69/js/vendor.chunk.js:1:188931)
    at XMLHttpRequest.n (https://cdn03.gitter.im/_s/280f48a69/js/vendor.chunk.js:1:926079)

Anytime one of these errors happens, the use gets a Unable to communicate with Gitter, We're having problems communicating with the server at the moment.... notification

---

2k events in frontend-staging

• Specific GATEWAY_TIMEOUT error
  • https://sentry.gitlab.net/gitlab/gitter-frontend-staging/issues/1096713/
  • https://sentry.gitlab.net/gitlab/gitter-frontend-staging/issues/1091011/
  • https://sentry.gitlab.net/gitlab/gitter-frontend-staging/issues/1090648/
• General 504
  • https://sentry.gitlab.net/gitlab/gitter-frontend-staging/issues/185924/?query=is:unresolved%20504

2 events in frontend

• Specific GATEWAY_TIMEOUT error
  • https://sentry.gitlab.net/gitlab/gitter-frontend/issues/1101917/
  • https://sentry.gitlab.net/gitlab/gitter-frontend/issues/1101903/
• If I just search for 504 in sentry there are some more things
  • https://sentry.gitlab.net/gitlab/gitter-frontend/issues/194442/
  • https://sentry.gitlab.net/gitlab/gitter-frontend/issues/195458/
  • https://sentry.gitlab.net/gitlab/gitter-frontend/issues/465476/

Also as reported by our users,

• https://gitter.zendesk.com/agent/tickets/13725
• @MadLittleMods has seen this on staging/next

---

In terms of endpoints, I have seen it on

• GET https://gitter.im/api_staging/private/issue-state?t=issue&p=gitlab&r=gitlab-org%2Fgitter%2Fwebapp&i=2364
• GET https://gitter.im/api_staging/v1/rooms/529c94dded5ab0b3bf04e16e/chatMessages?lookups%5B%5D=user&limit=101
• POST https://gitter.im/api_staging/v1/eyeballs

Cause

Unknown

Related info

The only 504 GATEWAY_TIMEOUT related stuff I know is we saw some when we mitigated https://gitlab.com/gitlab-org/gitter/webapp/issues/2291#white_check_mark-mitigation (2019-9-23)