GitHub oauth callbacks are frequently hanging for up to 5 minutes before failing with a 502

We are seeing requests to /users/auth/github/callback taking up to 300 seconds before failing.

image

https://log.gprd.gitlab.net/goto/7d2ff14ca11959cf8cb10f3d1bdb2708

Questions

  1. This endpoint makes an outgoing HTTP (TCP) call to GitHub. Could this be related to the NAT timeouts we're seeing at present? cc @hphilipps @craigf

  2. Why don't timeouts happen after (roughly) 60 seconds? cc @ayufan for puma timeouts knowledge.

  3. Would be possible to add an additional timeout on the oauth exchange (presumably this is something we could set in the HTTP client library we're using to do this exchange?) Anything over 30 seconds could probably be safely aborted? cc @lmcandrew for authentication

Incidents

Related to production#2324

Edited by Andrew Newdigate