Expand deployments blockers report
What does this MR do and why?
Update the deployment blockers report with instructions for release managers.
Related to gitlab-com/gl-infra/delivery#2536 (closed)
Author Check-list
-
Has documentation been updated?
Example
Click to expand
Overview
| Start date | End date | Production deployments blocked for | # of Production blockers |
|---|---|---|---|
| 2022-09-12 | 2022-09-18 | 0 | 0 |
Weekly overview
| Resource | Summary | Blocker type | gprd | gstg |
|---|
Additional incidents
Below is a list of production incidents created last week.
Click to expand
| Resource | Summary |
|---|---|
| gitlab-com/gl-infra/production#7756 (closed) | 2022-09-17: Patroni CI statement timeouts and elevated CI job error rate |
| gitlab-com/gl-infra/production#7755 (closed) | 2022-09-16: Prometheus notifications are queuing |
| gitlab-com/gl-infra/production#7754 (closed) | 2022-09-16: thanos restarting frequently |
| gitlab-com/gl-infra/production#7753 (closed) | 2022-09-16: CiRunnersServiceLoadbalancerErrorSLOViolation |
| gitlab-com/gl-infra/production#7752 (closed) | 2022-09-16: Degradation in some Thanos components |
| gitlab-com/gl-infra/production#7751 (closed) | 2022-09-16: Prometheus/Thanos rules failing and causing multiple alerts for all services |
| gitlab-com/gl-infra/production#7748 (closed) | 2022-09-15: The thanos_query_frontend SLI of the monitoring service (main stage) has an apdex violating SLO |
| gitlab-com/gl-infra/production#7747 (closed) | 2022-09-15: GKE Prometheus in us-east1-d is filling up its disk quickly |
| gitlab-com/gl-infra/production#7744 (closed) | 2022-09-15: DNS records for GKE Prometheus in gstg and pre missing |
| gitlab-com/gl-infra/production#7743 (closed) | 2022-09-14: Prometheus persistent volume filling up in cluster us-east1-d |
| gitlab-com/gl-infra/production#7742 (closed) | 2022-09-14: GCS snapshot failed for patroni-ci |
| gitlab-com/gl-infra/production#7741 (closed) | 2022-09-14: Pages sites with multiple domains experiencing Let's Encrypt issues |
| gitlab-com/gl-infra/production#7740 (closed) | 2022-09-14: Grafana LB error rate exceeding SLO |
| gitlab-com/gl-infra/production#7739 (closed) | 2022-09-14: Alertmanager failing to send notifications |
| gitlab-com/gl-infra/production#7738 (closed) | 2022-09-14: Alertmanager is failing sending notifications |
| gitlab-com/gl-infra/production#7737 (closed) | 2022-09-14: grafana_google_lb SLI of the monitoring service (main stage) has an error rate violating SLO |
| https://gitlab.com/gitlab-com/gl-infra/production/-/issues/7736 | 2022-09-14: Suspicion of a critical bug in pipeline processing |
| gitlab-com/gl-infra/production#7735 (closed) | 2022-09-14: Some frontend nodes appear to be down |
| gitlab-com/gl-infra/production#7734 (closed) | 2022-09-14: patroni-v12-10-db-gprd.c.gitlab-production.internal postgres service appears down |
| gitlab-com/gl-infra/production#7733 (closed) | 2022-09-14: Prometheus is backlogging on the notifications queue |
| gitlab-com/gl-infra/production#7729 (closed) | 2022-09-13: Websocket error rate spike briefly exceeded SLO |
| gitlab-com/gl-infra/production#7728 (closed) | 2022-09-13: LoggingVisibilityDiminished due to rails pubsub consumers saturating |
| gitlab-com/gl-infra/production#7727 (closed) | 2022-09-13: A slight uptick of web-pages 500 errors |
| gitlab-com/gl-infra/production#7724 (closed) | 2022-09-13: Prometheus PVC saturation in us-east1-b and us-east1-c |
| gitlab-com/gl-infra/production#7722 (closed) | 2022-09-12: WebsocketsServiceLoadbalancerErrorSLOViolation |
Instructions
-
Review the "Additional incidents" list and add the Deploys-blockedlabel if required. -
Update the Deployments metric review epic. -
Add a new row to the Overview section: Copy and paste the information from the Overview section in this issue. -
Fill out the Weekly Overview: Copy and paste the information from the Weekly overview in this issue. -
Update the Graph: Update the data on the spreadsheet and then update the graph on the epic.
-
Edited by Mayra Cabrera