OnCall report for period: 2018-07-10 - 2018-07-17
Oncall during this period
Schedule | Username |
---|---|
AMA | John Northrup |
EU | John Northrup |
EU | Ahmad Sherif |
EU | Jarv Jarvis |
PagerDuty Incidents
- Number of incidents: 25
Created | Summary |
---|---|
2018-07-11T00:24:02Z | [2387] Firing 5 - HighRailsErrorRate |
2018-07-11T03:19:31Z | [2388] Firing 5 - HighRailsErrorRate |
2018-07-11T04:02:03Z | [2389] Firing 7 - HighRailsErrorRate |
2018-07-11T04:04:42Z | [2390] Firing 1 - The alert test file is missing! |
2018-07-11T04:17:03Z | [2391] Firing 10 - HighRailsErrorRate |
2018-07-11T07:25:40Z | [2392] Pingdom check Dev.gitlab.org issue is down |
2018-07-11T07:41:04Z | [2393] Firing 4 - HighRailsErrorRate |
2018-07-11T11:33:35Z | [2394] Firing 4 - HighRailsErrorRate |
2018-07-11T13:22:10Z | [2395] Firing 5 - HighRailsErrorRate |
2018-07-11T14:41:36Z | [2396] Firing 9 - HighRailsErrorRate |
2018-07-11T14:44:05Z | [2397] Pingdom check GitLab Infrastructure Master Branch is down |
2018-07-11T14:45:02Z | [2398] Firing 6 - HighRailsErrorRate |
2018-07-11T14:47:58Z | [2399] Pingdom check GitLab.com Pages is down |
2018-07-11T15:03:10Z | [2400] Firing 1 - There are too many slow SQL queries |
2018-07-11T15:11:59Z | [2401] Firing 1 - There are too many slow SQL queries |
2018-07-11T15:15:45Z | [2402] Firing 5 - PostgreSQL_StatementTimeout_Errors |
2018-07-11T15:17:03Z | [2403] Firing 4 - HighRailsErrorRate |
2018-07-11T15:27:03Z | [2404] Firing 5 - HighRailsErrorRate |
2018-07-11T15:30:01Z | [2405] Pingdom check GitLab.com Pages is down |
2018-07-11T15:49:51Z | [2406] Firing 6 - HighRailsErrorRate |
2018-07-11T16:57:50Z | [2407] Firing 6 - HighRailsErrorRate |
2018-07-11T19:10:03Z | [2408] Firing 6 - HighRailsErrorRate |
2018-07-12T05:03:27Z | [2409] Firing 5 - HighRailsErrorRate |
2018-07-12T14:03:23Z | [2410] Firing 9 - HighRailsErrorRate |
2018-07-12T14:46:59Z | [2411] Pingdom check GitLab.com new repo is down |
Issues
7 Day OnCall Issue Stats
- Oncall issues : 10
- Access Request : 3
- Critical : 0
- Outage : 0
- Corrective Action : 1
Open OnCall Issue Stats
- Oncall issues : 23
- Access Request : 7
- Critical : 0
- Outage : 0
- Corrective Action : 15
Open Oncall Issues
Created | Assignee | Summary |
---|---|---|
16 Jul 18 06:16 UTC | unassigned | Maintenance on production postgres servers |
13 Jul 18 21:43 UTC | ddavison | VPN & SSH Access For ddavison |
13 Jul 18 14:30 UTC | unassigned | Fix meta refresh redirects |
13 Jul 18 13:19 UTC | jarv | revert the error rate relaxation |
11 Jul 18 13:25 UTC | unassigned | Configure and run the gitlab pseudonymizer |
11 Jul 18 10:18 UTC | unassigned | Running python script on a project created in the UI makes the project unavailable |
09 Jul 18 14:20 UTC | northrup | S3 Access for event tracking read/write |
09 Jul 18 11:53 UTC | ahmadsherif | Apply patch to fix helm tiller install |
09 Jul 18 06:39 UTC | unassigned | Gitaly p95 latency for nfs-02 is high |
04 Jul 18 01:27 UTC | unassigned | 2018-07-04 -- High CPU on nfs-file-07 |
04 Jul 18 00:59 UTC | unassigned | 2018-07-03 -- WAL-E Backup Delayed |
03 Jul 18 05:37 UTC | unassigned | Incorrect Storage Shard Key on Projects on GitLab.com |
28 Jun 18 14:20 UTC | unassigned | 2018-06-28: Multiple short outages affecting all GitLab services |
27 Jun 18 22:20 UTC | northrup | Read-only elastic cloud user for automated tasks |
21 Jun 18 23:12 UTC | unassigned | Environments not loading (502) |
15 Jun 18 11:24 UTC | unassigned | Remove Consul cluster for CI Monitoring from GCP part of the CI infrastructure |
13 Jun 18 02:38 UTC | unassigned | mtail fsnotify error |
12 Jun 18 21:00 UTC | unassigned | [www.design.gitlab.com] - Information Disclosure of system information |
08 Jun 18 23:13 UTC | unassigned | Misconfigured redirects/vhosts |
07 Jun 18 10:04 UTC | unassigned | Git-over-SSH operations were failing during 11.0-rc5 deploy |
30 May 18 13:13 UTC | unassigned | Invalid DKIM for failed pipeline email |
25 May 18 02:22 UTC | unassigned | Authentication errors in pipelines |
02 May 18 12:59 UTC | unassigned | Reports of 404s and repositories appearing empty |
Weekly Ops
Web/Git/API p95 latency
Gitaly p95 latency
Sidekiq CPU
API CPU
Git CPU
Web CPU
NFS timeouts
p50 Git latency for 200s
p95 Web latency for 200s
p95 Git latency for 200s
p50 API latency for 200s
p50 Web latency for 200s
p95 API latency for 200s
This issue was automatically generated using oncall-robot-assistant