Disk filling up on web-33-sv-gprd.c.gitlab-production.internal
Summary
Disk filling up on web-33-sv-gprd.c.gitlab-production.internal
Timeline
All times UTC.
2020-03-23
- 03:33 - Alert Fired https://gitlab.pagerduty.com/incidents/PJFJBW0?utm_source=slack&utm_campaign=channel
- 03:33 - @ggillies acknowledged the alert and started investigating
- 03:45 - @ggillies identified that disk was not full, but a lot of open file handles on deleted files
- 03:50 - @ggillies restarted
nginx
and observed that disk space usage dropped - 05:52 - Incident declared from Slack
Details
Disk almost full on web-33-sv-gprd.c.gitlab-production.internal
# df -h
Filesystem Size Used Avail Use% Mounted on
udev 30G 0 30G 0% /dev
tmpfs 5.9G 758M 5.2G 13% /run
/dev/sda1 20G 20G 62M 100% /
tmpfs 30G 4.0K 30G 1% /dev/shm
tmpfs 5.0M 0 5.0M 0% /run/lock
tmpfs 30G 0 30G 0% /sys/fs/cgroup
/dev/sdb 50G 2.8G 47G 6% /var/log
share-01-stor-gprd.c.gitlab-production.internal:/var/opt/gitlab/gitlab-rails/uploads 20T 12T 7.9T 60% /var/opt/gitlab/gitlab-rails/uploads
pages-01-stor-gprd.c.gitlab-production.internal:/var/opt/gitlab/gitlab-rails/shared/pages 16T 5.3T 11T 35% /var/opt/gitlab/gitlab-rails/shared/pages
share-01-stor-gprd.c.gitlab-production.internal:/var/opt/gitlab/gitlab-rails/shared/cache 20T 12T 7.9T 60% /var/opt/gitlab/gitlab-rails/shared/cache
share-01-stor-gprd.c.gitlab-production.internal:/var/opt/gitlab/gitlab-rails/shared/lfs-objects 20T 12T 7.9T 60% /var/opt/gitlab/gitlab-rails/shared/lfs-objects
share-01-stor-gprd.c.gitlab-production.internal:/var/opt/gitlab/gitlab-rails/shared/artifacts 20T 12T 7.9T 60% /var/opt/gitlab/gitlab-rails/shared/artifacts
share-01-stor-gprd.c.gitlab-production.internal:/var/opt/gitlab/gitlab-ci/builds 20T 12T 7.9T 60% /var/opt/gitlab/gitlab-ci/builds
tmpfs 5.9G 0 5.9G 0% /run/user/1055
Source
Incident declared by ggillies in Slack via /incident declare
command.
Resources
- If the Situation Zoom room was utilised, recording will be automatically uploaded to Incident room Google Drive folder (private)
Edited by Graeme Gillies