Sentry is down
Summary
Sentry is down
Timeline
All times UTC.
2020-05-12
- 07:47 - Sentry stops accepting requests
- 07:53 - Alert
sentry.gitlab.net is down
- 08:19 - Incident declared from Slack
- 08:33 - Sentry accepting connections again
- 08:34 - Alert resolves
Details
Nginx returns 504 - Gateway timeout
/var/log/supervisor/sentry-web.log
only contains endless "socket full" lines after the last working requests:
127.0.0.1 - - [11/May/2020:07:47:28 +0000] "POST /api/10/store/ HTTP/1.0" 200 366 "-" "sentry-ruby/2.9.0"
127.0.0.1 - - [11/May/2020:07:47:28 +0000] "POST /api/113/store/ HTTP/1.0" 200 366 "-" "Go-http-client/1.1"
169.254.169.254 - - [11/May/2020:07:47:28 +0000] "GET /auth/login/gitlab/ HTTP/1.1" 200 6002 "-" "GoogleHC/1.0"
127.0.0.1 - - [11/May/2020:07:47:28 +0000] "POST /api/12/store/ HTTP/1.0" 200 366 "-" "raven-go/1.0"
Mon May 11 07:47:43 2020 - *** uWSGI listen queue of socket "0.0.0.0:9000" (fd: 3) full !!! (101/100) ***
Mon May 11 07:47:44 2020 - *** uWSGI listen queue of socket "0.0.0.0:9000" (fd: 3) full !!! (101/100) ***
Mon May 11 07:47:45 2020 - *** uWSGI listen queue of socket "0.0.0.0:9000" (fd: 3) full !!! (101/100) ***
[...]
Source
Incident declared by hphilipps in Slack via /incident declare
command.
Resources
- If the Situation Zoom room was utilised, recording will be automatically uploaded to Incident room Google Drive folder (private)
Edited by Henri Philipps