2018-07-03 -- WAL-E Backup Delayed
The WAL-E backup delayed alarm went off and upon investigation, the WAL-E archiving process died because of insufficient space on the root disk:
2018-07-04_00:13:32.79720 postgres-02 postgresql: gpg: [stdout]: write error: No space left on device
2018-07-04_00:13:32.79756 postgres-02 postgresql: gpg: gpg: build_packet failed: file write error
2018-07-04_00:13:32.79785 postgres-02 postgresql: [stdout]: write error: No space left on device
2018-07-04_00:13:32.79831 postgres-02 postgresql: gpg: build_packet failed: file write error
2018-07-04_00:13:32.79990 postgres-02 postgresql: gpg: [stdout]: write error: No space left on device
2018-07-04_00:13:32.80009 postgres-02 postgresql: gpg: gpg: iobuf_flush failed on close: file write error
2018-07-04_00:13:32.80022 postgres-02 postgresql: [stdout]: write error: No space left on device
2018-07-04_00:13:32.80031 postgres-02 postgresql: gpg: iobuf_flush failed on close: file write error
2018-07-04_00:13:32.80045 postgres-02 postgresql: gpg: [stdin]: encryption failed: file write error
2018-07-04_00:13:32.80057 postgres-02 postgresql: gpg: [stdin]: encryption failed: file write error
2018-07-04_00:13:32.80074 postgres-02 postgresql: gpg: [stdout]: write error: No space left on device
2018-07-04_00:13:32.80086 postgres-02 postgresql: gpg: build_packet failed: file write error
2018-07-04_00:13:32.80103 postgres-02 postgresql: gpg: [stdout]: write error: No space left on device
2018-07-04_00:13:32.80141 postgres-02 postgresql: gpg: iobuf_flush failed on close: file write error
2018-07-04_00:13:32.80164 postgres-02 postgresql: gpg: [stdin]: encryption failed: file write errorUpon investigation, this was because /tmp was filled with several large files that began with tmp owned by the gitlab-psql user. I believe these to be tmp files as WAL-E encrypts and compresses the WAL files before upload (ls output below). Some of them were old and not open in lsof, so I deleted them. I left anything newer than 30 days.
After freeing some disk space, I killed the frozen lzop process. Since it exited with a non-zero code, WAL-E started again from that point. Thus, we lost no backups or any WAL files as a part of this alert.
We need to figure out why some of these files have stayed around and how to prevent this in the future.
I've removed unrelated files from the ls output for clarity.
root@postgres-02.db.prd.gitlab.com:/tmp# ls -l
total 7190672
-rw------- 1 gitlab-psql gitlab-psql 0 Jun 1 06:08 tmp1_27c9kv
-rw------- 1 gitlab-psql gitlab-psql 0 Jun 1 06:08 tmp2i5aotz0
-rw------- 1 gitlab-psql gitlab-psql 0 Jun 1 06:08 tmp2w3ity4c
-rw------- 1 gitlab-psql gitlab-psql 75383028 Apr 26 03:13 tmp3_iybkew
-rw------- 1 gitlab-psql gitlab-psql 0 Jun 1 03:47 tmp3qyrjwe9
-rw------- 1 gitlab-psql gitlab-psql 1132025672 Apr 26 03:11 tmp4en2o4wf
-rw------- 1 gitlab-psql gitlab-psql 0 Jun 1 06:08 tmp7ghq5oer
-rw------- 1 gitlab-psql gitlab-psql 0 Jun 1 06:08 tmp7_gntr43
-rw------- 1 gitlab-psql gitlab-psql 228833140 Jun 12 18:40 tmp7wjaevgh
-rw------- 1 gitlab-psql gitlab-psql 0 Jun 1 06:08 tmp84yd23lh
-rw------- 1 gitlab-psql gitlab-psql 0 Jun 1 06:08 tmp8shzh0td
-rw------- 1 gitlab-psql gitlab-psql 0 Jun 1 06:08 tmp8t7aj9nn
-rw------- 1 gitlab-psql gitlab-psql 0 Jun 1 06:08 tmpbc8538tr
-rw------- 1 gitlab-psql gitlab-psql 0 Jun 1 06:08 tmpbhoffx7u
-rw------- 1 gitlab-psql gitlab-psql 345981200 Jul 4 00:41 tmpd7uo0k12
-rw------- 1 gitlab-psql gitlab-psql 0 Jun 1 06:08 tmpf9hbtp17
-rw------- 1 gitlab-psql gitlab-psql 0 Jun 1 06:08 tmpfcjxisfx
-rw------- 1 gitlab-psql gitlab-psql 0 Jun 1 06:08 tmph6xabc1i
-rw------- 1 gitlab-psql gitlab-psql 0 Jun 1 03:47 tmph9_m3ts1
-rw------- 1 gitlab-psql gitlab-psql 0 Jun 1 06:08 tmpihh4bnyg
-rw------- 1 gitlab-psql gitlab-psql 0 Jun 1 06:08 tmpimjmfaaz
-rw------- 1 gitlab-psql gitlab-psql 979155148 Apr 26 03:12 tmpj0jhfnuk
-rw------- 1 gitlab-psql gitlab-psql 0 Jun 1 06:08 tmpj1sazwu_
-rw------- 1 gitlab-psql gitlab-psql 0 Jun 1 06:08 tmpjcs8o36h
-rw------- 1 gitlab-psql gitlab-psql 0 Jun 1 06:08 tmpjgpf2m66
-rw------- 1 gitlab-psql gitlab-psql 326459664 Jul 4 00:41 tmpkuyw0jq4
-rw------- 1 gitlab-psql gitlab-psql 978812759 Jun 12 18:40 tmpmcxmi3tx
-rw------- 1 gitlab-psql gitlab-psql 0 Jun 1 06:08 tmpnzn04hfe
-rw------- 1 gitlab-psql gitlab-psql 0 Jun 1 06:08 tmpofkwm48a
-rw------- 1 gitlab-psql gitlab-psql 0 Jun 1 06:08 tmpojbte_9c
-rw------- 1 gitlab-psql gitlab-psql 0 Jun 1 06:08 tmpp0koa1cc
-rw------- 1 gitlab-psql gitlab-psql 228463152 Jun 12 18:40 tmpp6ex8tfa
-rw------- 1 gitlab-psql gitlab-psql 0 Jun 1 06:08 tmpr3c8h6x9
-rw------- 1 gitlab-psql gitlab-psql 0 Jun 1 06:08 tmpr6d66elo
-rw------- 1 gitlab-psql gitlab-psql 0 Jun 1 06:08 tmpsoz3olbx
-rw------- 1 gitlab-psql gitlab-psql 237007557 Jun 12 18:40 tmpsq4d_7at
-rw------- 1 gitlab-psql gitlab-psql 32465168 Jul 4 00:41 tmpt7cqfdj3
-rw------- 1 gitlab-psql gitlab-psql 0 Jun 1 03:47 tmptk5gpr94
-rw------- 1 gitlab-psql gitlab-psql 0 Jun 1 06:08 tmptp6j7d03
-rw------- 1 gitlab-psql gitlab-psql 1181597696 Jul 4 00:16 tmpv1yju_qg
-rw------- 1 gitlab-psql gitlab-psql 0 Jun 1 03:47 tmpvthjo8ke
-rw------- 1 gitlab-psql gitlab-psql 0 Jun 1 06:08 tmpx61q7lxz
-rw------- 1 gitlab-psql gitlab-psql 0 Jun 1 06:08 tmpxa3id_vg
-rw------- 1 gitlab-psql gitlab-psql 0 Jun 1 06:08 tmpxgt8tnmd
-rw------- 1 gitlab-psql gitlab-psql 0 Jun 1 06:08 tmpxmt24er3
-rw------- 1 gitlab-psql gitlab-psql 981361121 Apr 26 03:13 tmpxucnuq8o
-rw------- 1 gitlab-psql gitlab-psql 0 Jun 1 06:08 tmpy3mss13m
-rw------- 1 gitlab-psql gitlab-psql 0 Jun 1 06:08 tmpztgotsyz
root@postgres-02.db.prd.gitlab.com:/tmp# file tmpy30qkchk
tmpy30qkchk: PGP RSA encrypted session key - keyid: <redacted> RSA (Encrypt or Sign) 2048b .cc/ @gl-infra