OnCall report for period: 2018-01-02 - 2018-01-09

Oncall during this period

Schedule	Username
AMA	Alex Hanselka
AMA	Jason Tevnan
EU	Ahmad Sherif
EU	Daniele Valeriani

PagerDuty Incidents

• Number of incidents: 25

Created	Summary
2018-01-02T23:14:50Z	[#1195] PostgreSQL replication slot with an stale xmin which can cause bloat on the primary
2018-01-02T23:39:51Z	[#1196 (closed)] PostgreSQL replication slot with an stale xmin which can cause bloat on the primary
2018-01-04T02:14:05Z	[#1197 (closed)] High Error Rate on Front End Web
2018-01-04T02:17:14Z	[#1198 (closed)] Pingdom check GitLab.com master branch is down
2018-01-04T02:17:17Z	[#1199 (moved)] Pingdom check GitLab.com new repo is down
2018-01-04T02:17:30Z	[#1200 (closed)] Pingdom check GitLab.com issue is down
2018-01-04T02:17:42Z	[#1201] Pingdom check GitLab.com public check is down
2018-01-04T02:19:46Z	[#1203] Postgres Replication lag is over 2 minutes
2018-01-04T02:19:46Z	[#1202 (closed)] Postgres Replication lag is over 2 minutes
2018-01-04T02:19:54Z	[#1204 (closed)] Postgres Replication lag is over 2 minutes
2018-01-04T02:19:54Z	[#1205 (closed)] Postgres Replication lag is over 2 minutes
2018-01-04T02:19:56Z	[#1206 (closed)] Postgres Replication lag is over 2 minutes
2018-01-04T02:19:56Z	[#1207] Postgres Replication lag is over 2 minutes
2018-01-04T02:19:56Z	[#1208 (closed)] Postgres Replication lag is over 2 minutes
2018-01-04T02:19:58Z	[#1209 (closed)] Postgres Replication lag is over 2 minutes
2018-01-04T02:25:29Z	[#1210 (closed)] Azure Reboot of Machines
2018-01-04T05:11:20Z	[#1211 (closed)] Git NFS Server nfs-file-08.stor.geo1.gitlab.com:9100 is down
2018-01-04T07:04:56Z	[#1212 (closed)] Pingdom check Static site is down
2018-01-04T07:04:59Z	[#1213] Pingdom check GitLab.org redirect is down
2018-01-04T07:05:42Z	[#1214 (closed)] Pingdom check GitLab.com public check is down
2018-01-06T06:21:10Z	[#1215 (closed)] No disk space left on /opt/prometheus/prometheus/data on prometheus-01.nyc1.do.gitlab-runners.gitlab.net: 0%
2018-01-06T08:33:11Z	[#1216 (closed)] No disk space left on /opt/prometheus/prometheus/data on prometheus-01.nyc1.do.gitlab-runners.gitlab.net: 0%
2018-01-06T10:27:10Z	[#1217 (closed)] No disk space left on /opt/prometheus/prometheus/data on prometheus-01.nyc1.do.gitlab-runners.gitlab.net: 0%
2018-01-07T18:12:04Z	[#1218 (moved)] Last WALE backup (from db3 to S3) was seen 1h 5m 31s ago.
2018-01-08T01:08:29Z	[#1219 (closed)] Postgres Replication lag is over 2 minutes

Issues

Stats for the last oncall period

• Total number of oncall issues opened in the last on call shift: 3
  • Access Request: 0
  • Critical: 1
• Total number of oncall issues closed in this milestone: 2
  • Access Request: 0
  • Critical: 1

Open OnCall Issues

• Total number of open oncall issues: 9
  • Access Request: 0
  • Critical: 1

Created	Assignee	Summary
08 Jan 18 02:17 UTC	ahanselka	PostgreSQL Split Brain
27 Dec 17 04:00 UTC	_stark	Postgresql replication lag over 200MB triggered
21 Dec 17 14:03 UTC	unassigned	Add alert for sequential reads
21 Dec 17 13:59 UTC	unassigned	Alert on errors in the pgbouncer log
21 Dec 17 11:33 UTC	tmaczukin	Add cleaning mechanism for runners-cache-X machines
21 Dec 17 09:25 UTC	zj	validate end-to-end artifact uploading on staging and enable on production
23 Nov 17 10:37 UTC	unassigned	Cleanup SSL certificates
15 Nov 17 10:34 UTC	unassigned	Detect and alarm on long-running orphan processes on sidekiq
06 Nov 17 14:30 UTC	unassigned	Alarms should go off when we fail to create azure snapshots

Weekly Ops

Web/Git/API p95 latency

Gitaly p95 latency

Sidekiq CPU

NFS timeouts

API CPU

Git CPU

Web CPU

This issue was automatically generated using oncall-robot-assistant

assigned to @ahmadsherif

closed