Weekly Reliability (SRE) Team Newsletter – On-call Period:2020-12-08 - 2020-12-15
This issue has been moved and the description cleared of content to avoid polluting the search results of this tracker, see the moved issue link for the original newsletter
Activity
added Reliability-Team-Newsletter label
marked this issue as related to #12041 (moved)
https://gitlab.com/gitlab-com/gl-infra/production/-/issues/3104 is being looked into. Needing to figure out how to make builds more reliable on handbook.
-
Notes from 2020-12-14 22:30 handover:
production#3185 (closed) - silence re-entered.
-
production#3174 (closed) is still ongoing and may alert fyi.
Wal-g backup alerts (last successful backup) had a silence and may go off again. Depends on the random node picked for doing the backup. Backups were running fine, but false alert due to timestamp on another node that was not cleared. Should not recur, but noting here.
Edited by Dave Smith 
https://ops.gitlab.net/gitlab-cookbooks/gitlab-walg/-/merge_requests/37 should fix the false alert for
walgBaseBackupDelayedafter a backup failure.
marked this issue as related to #12155 (moved)
added workflow-infraDone label
moved to reliability-reports#113 (closed)