2018-07-03 -- WAL-E Backup Delayed

The WAL-E backup delayed alarm went off and upon investigation, the WAL-E archiving process died because of insufficient space on the root disk:

2018-07-04_00:13:32.79720 postgres-02 postgresql: gpg: [stdout]: write error: No space left on device
2018-07-04_00:13:32.79756 postgres-02 postgresql: gpg: gpg: build_packet failed: file write error
2018-07-04_00:13:32.79785 postgres-02 postgresql: [stdout]: write error: No space left on device
2018-07-04_00:13:32.79831 postgres-02 postgresql: gpg: build_packet failed: file write error
2018-07-04_00:13:32.79990 postgres-02 postgresql: gpg: [stdout]: write error: No space left on device
2018-07-04_00:13:32.80009 postgres-02 postgresql: gpg: gpg: iobuf_flush failed on close: file write error
2018-07-04_00:13:32.80022 postgres-02 postgresql: [stdout]: write error: No space left on device
2018-07-04_00:13:32.80031 postgres-02 postgresql: gpg: iobuf_flush failed on close: file write error
2018-07-04_00:13:32.80045 postgres-02 postgresql: gpg: [stdin]: encryption failed: file write error
2018-07-04_00:13:32.80057 postgres-02 postgresql: gpg: [stdin]: encryption failed: file write error
2018-07-04_00:13:32.80074 postgres-02 postgresql: gpg: [stdout]: write error: No space left on device
2018-07-04_00:13:32.80086 postgres-02 postgresql: gpg: build_packet failed: file write error
2018-07-04_00:13:32.80103 postgres-02 postgresql: gpg: [stdout]: write error: No space left on device
2018-07-04_00:13:32.80141 postgres-02 postgresql: gpg: iobuf_flush failed on close: file write error
2018-07-04_00:13:32.80164 postgres-02 postgresql: gpg: [stdin]: encryption failed: file write error

Upon investigation, this was because /tmp was filled with several large files that began with tmp owned by the gitlab-psql user. I believe these to be tmp files as WAL-E encrypts and compresses the WAL files before upload (ls output below). Some of them were old and not open in lsof, so I deleted them. I left anything newer than 30 days.

After freeing some disk space, I killed the frozen lzop process. Since it exited with a non-zero code, WAL-E started again from that point. Thus, we lost no backups or any WAL files as a part of this alert.

We need to figure out why some of these files have stayed around and how to prevent this in the future.

I've removed unrelated files from the ls output for clarity.

root@postgres-02.db.prd.gitlab.com:/tmp# ls -l
total 7190672
-rw------- 1 gitlab-psql   gitlab-psql            0 Jun  1 06:08 tmp1_27c9kv
-rw------- 1 gitlab-psql   gitlab-psql            0 Jun  1 06:08 tmp2i5aotz0
-rw------- 1 gitlab-psql   gitlab-psql            0 Jun  1 06:08 tmp2w3ity4c
-rw------- 1 gitlab-psql   gitlab-psql     75383028 Apr 26 03:13 tmp3_iybkew
-rw------- 1 gitlab-psql   gitlab-psql            0 Jun  1 03:47 tmp3qyrjwe9
-rw------- 1 gitlab-psql   gitlab-psql   1132025672 Apr 26 03:11 tmp4en2o4wf
-rw------- 1 gitlab-psql   gitlab-psql            0 Jun  1 06:08 tmp7ghq5oer
-rw------- 1 gitlab-psql   gitlab-psql            0 Jun  1 06:08 tmp7_gntr43
-rw------- 1 gitlab-psql   gitlab-psql    228833140 Jun 12 18:40 tmp7wjaevgh
-rw------- 1 gitlab-psql   gitlab-psql            0 Jun  1 06:08 tmp84yd23lh
-rw------- 1 gitlab-psql   gitlab-psql            0 Jun  1 06:08 tmp8shzh0td
-rw------- 1 gitlab-psql   gitlab-psql            0 Jun  1 06:08 tmp8t7aj9nn
-rw------- 1 gitlab-psql   gitlab-psql            0 Jun  1 06:08 tmpbc8538tr
-rw------- 1 gitlab-psql   gitlab-psql            0 Jun  1 06:08 tmpbhoffx7u
-rw------- 1 gitlab-psql   gitlab-psql    345981200 Jul  4 00:41 tmpd7uo0k12
-rw------- 1 gitlab-psql   gitlab-psql            0 Jun  1 06:08 tmpf9hbtp17
-rw------- 1 gitlab-psql   gitlab-psql            0 Jun  1 06:08 tmpfcjxisfx
-rw------- 1 gitlab-psql   gitlab-psql            0 Jun  1 06:08 tmph6xabc1i
-rw------- 1 gitlab-psql   gitlab-psql            0 Jun  1 03:47 tmph9_m3ts1
-rw------- 1 gitlab-psql   gitlab-psql            0 Jun  1 06:08 tmpihh4bnyg
-rw------- 1 gitlab-psql   gitlab-psql            0 Jun  1 06:08 tmpimjmfaaz
-rw------- 1 gitlab-psql   gitlab-psql    979155148 Apr 26 03:12 tmpj0jhfnuk
-rw------- 1 gitlab-psql   gitlab-psql            0 Jun  1 06:08 tmpj1sazwu_
-rw------- 1 gitlab-psql   gitlab-psql            0 Jun  1 06:08 tmpjcs8o36h
-rw------- 1 gitlab-psql   gitlab-psql            0 Jun  1 06:08 tmpjgpf2m66
-rw------- 1 gitlab-psql   gitlab-psql    326459664 Jul  4 00:41 tmpkuyw0jq4
-rw------- 1 gitlab-psql   gitlab-psql    978812759 Jun 12 18:40 tmpmcxmi3tx
-rw------- 1 gitlab-psql   gitlab-psql            0 Jun  1 06:08 tmpnzn04hfe
-rw------- 1 gitlab-psql   gitlab-psql            0 Jun  1 06:08 tmpofkwm48a
-rw------- 1 gitlab-psql   gitlab-psql            0 Jun  1 06:08 tmpojbte_9c
-rw------- 1 gitlab-psql   gitlab-psql            0 Jun  1 06:08 tmpp0koa1cc
-rw------- 1 gitlab-psql   gitlab-psql    228463152 Jun 12 18:40 tmpp6ex8tfa
-rw------- 1 gitlab-psql   gitlab-psql            0 Jun  1 06:08 tmpr3c8h6x9
-rw------- 1 gitlab-psql   gitlab-psql            0 Jun  1 06:08 tmpr6d66elo
-rw------- 1 gitlab-psql   gitlab-psql            0 Jun  1 06:08 tmpsoz3olbx
-rw------- 1 gitlab-psql   gitlab-psql    237007557 Jun 12 18:40 tmpsq4d_7at
-rw------- 1 gitlab-psql   gitlab-psql     32465168 Jul  4 00:41 tmpt7cqfdj3
-rw------- 1 gitlab-psql   gitlab-psql            0 Jun  1 03:47 tmptk5gpr94
-rw------- 1 gitlab-psql   gitlab-psql            0 Jun  1 06:08 tmptp6j7d03
-rw------- 1 gitlab-psql   gitlab-psql   1181597696 Jul  4 00:16 tmpv1yju_qg
-rw------- 1 gitlab-psql   gitlab-psql            0 Jun  1 03:47 tmpvthjo8ke
-rw------- 1 gitlab-psql   gitlab-psql            0 Jun  1 06:08 tmpx61q7lxz
-rw------- 1 gitlab-psql   gitlab-psql            0 Jun  1 06:08 tmpxa3id_vg
-rw------- 1 gitlab-psql   gitlab-psql            0 Jun  1 06:08 tmpxgt8tnmd
-rw------- 1 gitlab-psql   gitlab-psql            0 Jun  1 06:08 tmpxmt24er3
-rw------- 1 gitlab-psql   gitlab-psql    981361121 Apr 26 03:13 tmpxucnuq8o
-rw------- 1 gitlab-psql   gitlab-psql            0 Jun  1 06:08 tmpy3mss13m
-rw------- 1 gitlab-psql   gitlab-psql            0 Jun  1 06:08 tmpztgotsyz

root@postgres-02.db.prd.gitlab.com:/tmp# file tmpy30qkchk
tmpy30qkchk: PGP RSA encrypted session key - keyid: <redacted> RSA (Encrypt or Sign) 2048b .

cc/ @gl-infra

Assignee Loading
Time tracking Loading