2020-02-10: Spawn timeouts and Gitaly errors on file-cny-01
Summary
More information will be added as we investigate the issue.
Timeline
All times UTC.
2020-02-10
- 09:10 - Gitaly spawn timeouts detected on file-cny-01 (the only file-cny node): https://dashboards.gitlab.net/d/gitaly-host-detail/gitaly-host-detail?orgId=1&from=1581323615642&to=1581326799868&var-PROMETHEUS_DS=Global&var-environment=gprd&var-fqdn=file-cny-01-stor-gprd.c.gitlab-production.internal
- 09:21 - SRE on-call alerted to error ratio in gitaly's cny stage exceeding that of the SLO, while the main stage is healthy.
Edited by Craig Furman