On-Call Handover 2021-04-28 15:00 UTC
On-Call Handover
Brought to you by the Slack slash command: /sre-oncall handover
- EOC egress: @ahmadsherif
- EOC ingress: @nnelson
Summary:
A mostly-quiet shift. We tried enabling gitlab-org/gitlab#326095 (closed) but it didn't work as planned so we reverted it, got a page as a result. Two other pages related to the database, they resolved on their own.
What (if any) time-critical work is being handed over?
What contextual info may be useful for the next few on-call shifts?
Ongoing alerts/incidents:
-
production#4379 (closed) - 2021-04-28: Prometheus has slow rule evaluations
-
production#4378 (closed) - 2021-04-28: Prometheus has slow rule evaluations
-
production#4375 (closed) - 2021-04-28: SSL certificate for sytses/test-2#1 expires soon
-
production#4374 (closed) - 2021-04-28: SSL certificate for https://gitlab.com/gitlab-com/www-gitlab-com expires soon
-
production#4373 (closed) - 2021-04-28: SSL certificate for https://gitlab.com/gitlab-com/gl-infra/infrastructure/-/issues/57 expires soon
-
production#4372 (closed) - 2021-04-28: SSL certificate for sytses/test-2#1 expires soon
-
production#4371 (closed) - 2021-04-28: SSL certificate for https://gitlab.com/explore expires soon
-
production#4369 (closed) - 2021-04-28: SSL certificate for https://gitlab.com/gitlab-com/infrastructure/issues expires soon
-
production#4370 (closed) - 2021-04-28: SSL certificate for https://gitlab.com/explore/projects/starred expires soon
-
production#4366 (closed) - 2021-04-28: SSL certificate for https://gitlab.com/api/v4/projects/gitlab-org%2Fgitlab-foss expires soon
-
production#4368 (closed) - 2021-04-28: SSL certificate for https://www.gitlab.com expires soon
-
production#4365 (closed) - 2021-04-28: SSL certificate for https://gitlab.com/projects/new expires soon
-
production#4364 (closed) - 2021-04-28: SSL certificate for https://gitlab.com expires soon
-
production#4363 (closed) - 2021-04-28: SSL certificate for https://gitlab.com/gitlab-org/gitlab-foss/tree/master expires soon
-
production#4367 (closed) - 2021-04-28: SSL certificate for https://gitlab.com/gitlab-org/gitlab-foss/merge_requests/ expires soon
-
production#4361 (closed) - 2021-04-28: SSL certificate for gitlab-org/gitlab-foss#1 (closed) expires soon
-
production#4362 (closed) - 2021-04-28: SSL certificate for https://gitlab.com/gitlab-org/gitlab-foss/issues expires soon
-
production#4360 (closed) - 2021-04-28: SSL certificate for https://gitlab.com/gitlab-com/gl-infra/infrastructure/raw/master/alert-test expires soon
-
production#4359 (closed) - 2021-04-28: SSL certificate for https://gitlab.com/gitlab-com/gitlab-com-infrastructure/tree/master expires soon
-
production#4358 (closed) - 2021-04-28: SSL certificate for https://gitlab.com/gitlab-org/gitlab-foss/ expires soon
Resolved actionable alerts:
-
https://gitlab.pagerduty.com/incidents/PM3FV0T - [#41509] Firing 1 - The Redis Primary CPU Utilization per Node resource of the redis-sidekiq service (main stage), component has a saturation exceeding SLO and is close to its capacity limit.
-
https://gitlab.pagerduty.com/incidents/PP8HYBU - [#41512] Firing 1 - Transactions detected that have been running on
patroni-03-db-gprd.c.gitlab-production.internalfor more than 10m