Skip to content
Snippets Groups Projects

Weekly Reliability (SRE) Team Newsletter – On-call Period: 2024-02-06 - 2024-02-13

    • Closed Issue created by ops-gitlab-net @ops-gitlab-net

    Announcements

    Engineering Week in Review Highlights:

    Team Updates

    On-Call During This Period

    Schedule Username
    SRE 8-hour Americas Alex Hanselka
    SRE 8-hour Americas Matt Smiley
    SRE 8-hour APAC Devin Sylva
    SRE 8-hour APAC Pierre Guinoiseau
    SRE 8-hour EMEA Igor Wiedler
    SRE 8-hour EMEA Maina Ng'ang'a
    SRE 8-hour EMEA Calliope Gardner

    PagerDuty Incidents

    See the 1 week report for acknowledged PD pages (long-term trend)

    Alerts Volume

    7 Day Issue Stats

    • Oncall issues : 0
    • Access Request : 0
    • Change Issues : 5
    • Incident Issues : 17
    • CorrectiveAction Issues : 0

    Change Issues

    Incident Issues

    CorrectiveAction Issues

    Open Issue Stats

    Open Change Issues

    Show/Hide Table
    Created Summary
    2024-02-12T06:55:53Z 2024-02-12: Set arkose_labs_data_exchange_key a... (production#17567 - closed)
    2024-02-06T22:56:52Z GPRD - Implement data checksums in a GPRD Main ... (production#17544 - closed)
    2024-02-05T18:12:15Z 2024-02-05: [gprd] Combine redundant skip rules... (production#17534 - closed)
    2024-02-02T02:32:58Z https://gitlab.com/gitlab-com/gl-infra/production/-/issues/17521+
    2024-01-31T08:56:33Z https://gitlab.com/gitlab-com/gl-infra/production/-/issues/17507+
    2024-01-24T13:17:29Z Renew TLS certificate manually for airflow.gitl... (production#17463 - closed)
    2024-01-23T08:24:06Z 2024-02-01: Query Git repos blob sizes (production#17450 - closed)
    2024-01-22T13:47:27Z [GPRD] Implement data checksum for GPRD CI clus... (production#17441 - closed)
    2024-01-22T03:24:34Z 2024-01-22: Change Staging GitHub OAuth app det... (production#17435 - closed)
    2024-01-19T20:10:14Z 2024-01-19: 6 corrupt test projects on staging ... (production#17431 - closed)
    2024-01-17T05:33:36Z 2024-02-14 Update GitLab.com deployment to use ... (production#17411 - closed)
    2024-01-16T07:41:03Z https://gitlab.com/gitlab-com/gl-infra/production/-/issues/17401+
    2024-01-15T05:49:11Z Retroactive/informational change - Wiz.io usage... (production#17397 - closed)
    2024-01-10T13:38:03Z Change Request [STG] Run rake task `gitlab:clea... (production#17383 - closed)
    2023-12-14T12:12:09Z 2023-12-14: Enable running ClickHouse migration... (production#17278 - closed)
    2023-12-14T09:36:10Z 2023-12-20: [GSTG] Official gameday (production#17274 - closed)
    2023-12-08T23:04:03Z https://gitlab.com/gitlab-com/gl-infra/production/-/issues/17239+
    2023-12-08T22:59:06Z Add more disk storage to production Global Sear... (production#17238 - closed)
    2023-12-04T12:01:23Z 2024-02-22: Add OSQuery to Monitoring Prod/Ops ... (production#17214 - closed)
    2023-11-30T17:58:45Z 2023-11-30: Upgrade file-cny-01-stor-gprd machi... (production#17198 - closed)
    2023-10-25T23:47:08Z [GSTG] - Upgrade pgbouncers to version 1.21 (production#17036 - closed)
    2023-10-24T15:09:17Z Run Clean up deleted merge requests for the git... (production#17023 - closed)
    2023-10-18T12:15:40Z https://gitlab.com/gitlab-com/gl-infra/production/-/issues/16997+
    2023-08-17T08:56:24Z [GPRD] Destroy `patroni-v12-ci` cluster (production#16189 - closed)
    2023-08-10T07:26:41Z https://gitlab.com/gitlab-com/gl-infra/production/-/issues/16156+
    2023-07-31T03:10:25Z [GPRD] - Restart VMs to fix the issue related t... (production#16099 - closed)
    2023-07-19T17:38:59Z 2023-07-TBD: [DRAFT] [GPRD] - Delete unused Pat... (production#16062 - closed)
    2023-06-27T10:25:48Z Draft: [GPRD] PgBouncer upgrade (production#15939 - closed)
    2023-06-27T10:24:13Z Draft: [GSTG] PgBouncer upgrade (production#15938 - closed)
    2023-06-27T10:22:54Z [db-benchmarking] PgBouncer upgrade (production#15937 - closed)
    2023-05-18T08:07:10Z [GPRD] Install pg_stat_kcache package in DR pos... (production#14449 - closed)
    2023-05-04T02:19:45Z 2023-05-04: Remove admin rights from ops-gitlab... (production#11074 - closed)
    2023-03-28T16:11:57Z [GPRD] Execute logical replication and upgrade ... (production#8611 - closed)
    2022-12-22T03:57:59Z 2022-12-22: Add registry expiration policies to... (production#8186 - closed)
    2022-11-14T23:01:34Z Change the schedule of automatic database reind... (production#8048 - closed)
    2022-10-18T07:03:19Z 2022-10-18: [GPRD] Use turbo mode on restore co... (production#7893 - closed)
    2022-10-18T06:45:39Z 2022-10-18: [GPRD] Use turbo mode on restore co... (production#7892 - closed)
    2022-09-02T09:37:55Z Removal of sigin-Page text for staging and .com (production#7681 - closed)

    Open Incident Issues

    Show/Hide Table
    Created Summary
    2024-02-11T11:02:15Z 2024-02-11: PostgreSQL queries dominating total... (production#17563 - closed)
    2024-01-31T09:36:14Z 2024-01-31: password authentication failed for ... (production#17509 - closed)

    Open Oncall Issues

    Show/Hide Table
    Created Summary
    2021-09-17T19:35:34Z Proposal: When an Incident is declared, output ... (production-engineering#14205)
    2020-12-18T22:29:14Z CI clones fail for repositories with a path end... (production-engineering#12200 - moved)

    Issues for Review during Incident Review Meeting

    If there are any incidents you think would be good to review, please add them to the Agenda for the next meeting.
    Edited by ops-gitlab-net
    To upload designs, you'll need to enable LFS and have an admin enable hashed storage. More information

    Child items ...

    • Activity

    Please register or sign in to reply