OnCall report for period: 2018-03-13 - 2018-03-20
Oncall during this period
Schedule | Username |
---|---|
AMA | Alejandro Rodriguez |
AMA | Ilya Frolov |
EU | Ahmad Sherif |
EU | Jason Tevnan |
PagerDuty Incidents
- Number of incidents: 25
Created | Summary |
---|---|
2018-03-14T14:50:18Z | [1387] Pingdom check GitLab.com master branch is down |
2018-03-14T14:55:15Z | [1388] Pingdom check GitLab.com master branch is down |
2018-03-14T17:25:31Z | [1389] Pingdom check GitLab.com new repo is down |
2018-03-14T17:26:56Z | [1390] Pingdom check GitLab.com master branch is down |
2018-03-14T17:27:34Z | [1391] Pingdom check GitLab.com issue is down |
2018-03-14T17:27:43Z | [1392] Pingdom check GitLab.com public check is down |
2018-03-15T03:48:38Z | [1393] Firing 1 - CPU use percent is extremely high on sidekiq-pullmirror-02.sv.prd.gitlab.com for the past 2 hours. |
2018-03-15T09:17:37Z | [1394] Firing 1 - 1% disk space left |
2018-03-15T23:16:40Z | [1395] Pingdom check Dev.gitlab.org issue is down |
2018-03-16T20:11:45Z | [1396] Pingdom check Dev.gitlab.org issue is down |
2018-03-16T21:06:40Z | [1397] Pingdom check Dev.gitlab.org issue is down |
2018-03-18T07:26:42Z | [1398] Pingdom check Dev.gitlab.org issue is down |
2018-03-19T13:09:11Z | [1399] Pingdom check GitLab.com Pages is down |
2018-03-19T14:13:48Z | [1401] Pingdom check GitLab.org redirect is down |
2018-03-19T14:13:48Z | [1400] Pingdom check Static site is down |
2018-03-19T14:13:51Z | [1402] Pingdom check GitLab.com public check is down |
2018-03-19T15:15:54Z | [1403] Pingdom check GitLab.com master branch is down |
2018-03-19T16:27:19Z | [1404] Pingdom check GitLab.com new repo is down |
2018-03-19T16:27:23Z | [1405] Pingdom check GitLab.com master branch is down |
2018-03-19T16:27:43Z | [1406] Pingdom check GitLab.com public check is down |
2018-03-19T16:27:46Z | [1407] Pingdom check GitLab.com issue is down |
2018-03-19T16:28:19Z | [1408] Firing 1 - High Error Rate on Front End Web |
2018-03-19T16:28:30Z | [1409] Firing 1 - Postgres seems to be consuming transaction IDs very slowly |
2018-03-19T16:29:31Z | [1410] Firing 4 - Postgres seems to be processing very few transactions |
2018-03-19T18:10:25Z | [1411] Pingdom check GitLab.com master branch is down |
Issues
7 Day OnCall Issue Stats
- Oncall issues : 13
- Access Request : 2
- Critical : 0
- Outage : 0
- Corrective Action : 3
Open OnCall Issue Stats
- Oncall issues : 14
- Access Request : 4
- Critical : 0
- Outage : 0
- Corrective Action : 17
Open Oncall Issues
Created | Assignee | Summary |
---|---|---|
20 Mar 18 12:12 UTC | unassigned | gitlab-sidekiq alerting generating regular alerts |
19 Mar 18 14:34 UTC | ahmadsherif | about.gitlab.com was down briefly on March 19th |
19 Mar 18 12:03 UTC | unassigned | Mystery AWS access key, might want to revoke? |
16 Mar 18 16:55 UTC | unassigned | SSH access to customers and license apps |
16 Mar 18 15:22 UTC | unassigned | Turn on repository verification checksum feature |
16 Mar 18 11:24 UTC | unassigned | nfs-file-07 load spiked up to ~150 |
15 Mar 18 23:47 UTC | nolith | March 15th dev.gitlab.org outage |
15 Mar 18 12:39 UTC | unassigned | Transfer of Gemnasium domains |
13 Mar 18 18:45 UTC | unassigned | API request latency for Status 200 requests is all over the show |
12 Mar 18 11:22 UTC | northrup | The new pricing changes redirects, "purged" the /gitlab-com/settings page |
09 Mar 18 23:36 UTC | unassigned | Use /etc/gitlab/skip-auto-reconfigure and remove /etc/gitlab/skip-auto-migrations |
07 Mar 18 12:44 UTC | unassigned | Chef and SSH access request for Filipa Lacerda |
06 Mar 18 17:51 UTC | unassigned | Chef and SSH access request for Mayra Cabrera |
12 Feb 18 08:51 UTC | bjk-gitlab | Re-enable NFS metrics collection in node_exporter |
Weekly Ops
Web/Git/API p95 latency
p50 Git latency for 200s
Gitaly p95 latency
p95 Git latency for 200s
p95 Web latency for 200s
API CPU
p50 Web latency for 200s
p95 API latency for 200s
Web CPU
p50 API latency for 200s
Sidekiq CPU
Git CPU
NFS timeouts
This issue was automatically generated using oncall-robot-assistant