Issues · GitLab.com / GitLab Infrastructure Team / production · GitLab

2021-07-28: Frequent dead man's snitch failures on database backup non-existence

#5245 · created Jul 28, 2021 by Nels Nelson 1 IncidentMitigated RootCauseExternal-Dependency ServicePostgres incident severity4
- 12
updated Sep 05, 2021
2021-09-03 - Staging deploys blocked on Kubernetes rollout 0 of 3 tasks completed

#5480 · created Sep 03, 2021 by ops-gitlab-net Deploys-blocked-gstg2hr IncidentMitigated NeedsRootCause ServiceSidekiq Source::IMAIncidentDeclare incident incident-typedeployment related severity2
- 3
updated Sep 05, 2021
2021-09-03: Apdex dips on Gitaly node file-43 0 of 3 tasks completed

#5477 · created Sep 03, 2021 by ops-gitlab-net IncidentActive NeedsRootCause ServiceGitaly Source::IMAIncidentDeclare incident severity3
- 8
updated Sep 05, 2021
2021-09-03: The imagescaler SLI of the web service in region `us-east1-d` has an apdex violating SLO 0 of 3 tasks completed

#5474 · created Sep 03, 2021 by ops-gitlab-net IncidentMitigated NeedsRootCause ServiceWeb Source::IMAIncidentDeclare alertmanager-silence incident severity3
- 9
updated Sep 06, 2021
2021-09-02: Prometheus not connected to any Alertmanagers 0 of 3 tasks completed

#5466 · created Sep 02, 2021 by ops-gitlab-net IncidentMitigated RootCauseSPoF ServicePrometheus Source::IMAIncidentDeclare alertmanager-silence incident severity3
- 41
updated Sep 06, 2021
2021-09-02: Apdex dip on Gitaly node file-praefect-02 0 of 3 tasks completed

#5462 · created Sep 02, 2021 by ops-gitlab-net IncidentMitigated NeedsRootCause ServiceGitaly Source::IMAIncidentDeclare incident severity3
- 10
updated Sep 05, 2021
2021-08-30: The gitlab_zone SLI of the waf service (`main` stage) has an error rate violating SLO 0 of 3 tasks completed

#5438 · created Aug 30, 2021 by ops-gitlab-net IncidentMitigated NeedsRootCause ServiceFrontend Source::IMAIncidentDeclare alertmanager-silence incident severity3
- 16
updated Sep 06, 2021
2021-08-26: The workhorse SLI of the git service in region `us-east1-b` has an apdex violating SLO 0 of 3 tasks completed

#5429 · created Aug 27, 2021 by ops-gitlab-net IncidentMitigated NeedsRootCause ServiceGit Source::IMAIncidentDeclare incident severity3
- 25
updated Sep 05, 2021
2021-08-26: Firing 1 - The gitlab_zone SLI of the waf service (`main` stage) has an error rate violating SLO 0 of 3 tasks completed

#5424 · created Aug 26, 2021 by ops-gitlab-net IncidentMitigated RootCauseMalicious-Traffic ServiceCloudflare Source::IMAIncidentDeclare incident severity3
- 6
updated Sep 05, 2021
2021-08-13: Alertmanager is failing sending notifications 0 of 3 tasks completed

#5352 · created Aug 13, 2021 by ops-gitlab-net IncidentMitigated RootCauseSaturation ServiceMonitoring Source::IMAIncidentDeclare alertmanager-silence incident severity4
- 5
updated Sep 05, 2021
2021-07-15: The goserver_op_service SLI of the gitaly service on node `file-27-stor-gprd.c.gitlab-production.internal` has an apdex violating SLO

#5172 · created Jul 15, 2021 by ops-gitlab-net IncidentMitigated RootCauseMalicious-Traffic ServiceGitaly Source::IMAIncidentDeclare alertmanager-silence incident severity3
- 36
updated Sep 05, 2021
2021-07-02: Apdex spike on Gitaly node file-praefect-02

#5066 · created Jul 02, 2021 by ops-gitlab-net IncidentMitigated RootCauseSaturation ServiceGitaly Source::IMAIncidentDeclare incident severity3
- 8
updated Sep 05, 2021
2021-07-01: Apdex and error rate spikes on 3 praefect-fronted gitaly nodes

#5057 · created Jul 01, 2021 by ops-gitlab-net IncidentMitigated RootCauseSaturation ServicePraefect Source::IMAIncidentDeclare incident severity3
- 20
updated Sep 05, 2021