Weekly Reliability (SRE) Team Newsletter – On-call Period: 2022-11-08 - 2022-11-15

Announcements

Engineering Week in Review Highlights:

Team Updates

On-Call During This Period

Schedule Username
SRE 8-hour Americas Alex Hanselka
SRE 8-hour Americas Hendrik Meyer
SRE 8-hour APAC Devin Sylva
SRE 8-hour EMEA Alejandro Rodríguez
SRE 8-hour EMEA Igor Wiedler
SRE 8-hour EMEA Steve Azzopardi

PagerDuty Incidents

See the 1 week report for acknowledged PD pages (long-term trend)

Alerts Volume

7 Day Issue Stats

  • Oncall issues : 0
  • Access Request : 0
  • Change Issues : 8
  • Incident Issues : 18
  • CorrectiveAction Issues : 0

Change Issues

Incident Issues

CorrectiveAction Issues

Open Issue Stats

Open Change Issues

Show/Hide Table
Created Summary
2022-11-14T00:28:30Z 2022-11-28: [GPRD] Upgrade Consul agents on rem... (production#8041 - closed)
2022-11-14T00:08:46Z 2022-11-22: [GPRD] Update Consul agent on Patro... (production#8040 - closed)
2022-11-11T06:39:53Z 2022-11-21: [GPRD] Upgrade Consul agents in k8s... (production#8038 - closed)
2022-11-11T06:29:28Z 2022-11-16: [GPRD] Upgrade Consul cluster to 1.... (production#8037 - closed)
2022-11-11T04:20:52Z 2022-11-15: GPRD: Roll out max_replica_pools to... (production#8036 - closed)
2022-11-10T06:25:27Z 2022-11-16: [GPRD] Deploy Consul cluster in k8s... (production#8032 - closed)
2022-11-01T17:39:50Z 2022-11-21 (C2) Automate dev.gitlab.org and reg... (production#7976 - closed)
2022-11-01T10:03:04Z Rollout logical replication in staging (production#7969 - closed)
2022-11-01T00:52:47Z CR - GPRD - Upgrade Ubuntu on PGBouncer nodes (production#7967 - closed)
2022-11-01T00:52:32Z CR - GSTG - Upgrade Ubuntu on PGBouncer nodes (production#7966 - closed)
2022-10-28T17:02:10Z Update instance-wide OAuth Application on GitLa... (production#7942 - closed)
2022-10-25T13:23:59Z [PRDSUB] Migrate to CloudSQL (production#7922 - closed)
2022-10-18T07:03:19Z 2022-10-18: [GPRD] Use turbo mode on restore co... (production#7893 - closed)
2022-10-18T06:45:39Z 2022-10-18: [GPRD] Use turbo mode on restore co... (production#7892 - closed)
2022-10-14T03:26:01Z https://gitlab.com/gitlab-com/gl-infra/production/-/issues/7877+
2022-10-11T15:17:53Z 2022-12-01: Gradually increase the number of ma... (production#7862 - closed)
2022-10-07T22:14:21Z Delete the production gemnasium Cloud SQL datab... (production#7848 - closed)
2022-10-06T07:05:45Z Disable fastupdate on merge_requests GIN indexes (production#7840 - closed)
2022-10-05T01:45:36Z 2022-10-05: Rollout PREVENT_LOAD_BALANCER_RETRI... (production#7836 - closed)
2022-10-05T00:00:51Z Reduce io-threads on redis-persistent in staging (production#7834 - closed)
2022-09-23T15:35:33Z Fill in DORA configuration for gitlab-org/gitla... (production#7793 - closed)
2022-09-02T09:37:55Z Removal of sigin-Page text for staging and .com (production#7681 - closed)
2022-08-25T14:11:48Z [GPRD] Enable pg_wait_sampling in Production (production#7653 - closed)
2022-08-17T19:40:56Z Move version.gitlab.com to new namespace, GCP p... (production#7615 - closed)
2022-08-16T16:43:58Z 2022-08-16: Increase max_client_conn for pgboun... (production#7607 - closed)
2022-07-22T14:23:23Z 2022-07-22: Update home page url in staging fro... (production#7497 - closed)
2022-06-06T15:55:01Z Rollout usage of ci-gateway ILB for on shared r... (production#7206 - closed)
2022-05-23T16:22:23Z 2022-05-23: Update auth scopes to include servi... (production#7117 - closed)

Open Incident Issues

Show/Hide Table
Created Summary
2022-11-10T15:55:55Z 2022-11-10: Increased 502 errors on api/v4/jobs... (production#8033 - closed)
2022-11-07T08:53:23Z 2022-11-07: SSL certificate for dev.gitlab.org:... (production#8009 - closed)
2022-11-07T04:16:15Z https://gitlab.com/gitlab-com/gl-infra/production/-/issues/8006+
2022-11-06T21:53:45Z https://gitlab.com/gitlab-com/gl-infra/production/-/issues/8005+
2022-11-02T14:18:07Z 2022-11-02: Intermittent Internal API unreachable (production#7979 - closed)
2022-10-29T13:50:58Z 2022-10-29: Chef client has been disabled for a... (production#7947 - closed)
2022-10-25T20:05:24Z 2022-10-25: Intermittent kas.gitlab.com timeouts (production#7924 - closed)

Open Oncall Issues

Show/Hide Table
Created Summary
2021-09-17T19:35:34Z https://gitlab.com/gitlab-com/gl-infra/reliability/-/issues/14205+
2020-12-18T22:29:14Z https://gitlab.com/gitlab-com/gl-infra/reliability/-/issues/12200+

Issues for Review during Incident Review Meeting

If there are any incidents you think would be good to review, please add them to the Agenda for the next meeting.
Edited by ops-gitlab-net