*WIP* CR for fixing backup logging

Production Change - Criticality 1 C1

Change Objective Describe the objective of the change
Change Type Avoid overwrite of backup's log file on every run
Services Impacted wal-e
Change Team Members @gerardo.herzig
Change Severity C1
Buddy check @gerardo.herzig
Tested in staging
Schedule of the change 22/11/2019 00:00 UTC
Duration of the change 10 minutes
Downtime Component no
Detailed steps for the change. Each step must include: see bellow

From #1243 (comment 229087284), one of the action items is to avoid wal-e backups to overwrite its own logfile in every call. As this logfile will grow pretty fast (2-3 MB per day), we should incorporate it into the logrotate schema.

An entry for wal-e already exists in logrotate.d:

 root@patroni-02-db-gprd.c.gitlab-production.internal:/etc/logrotate.d# cat wal-e 
/var/log/wal-e/wal-e.log {
  daily
  missingok
  rotate 7
  compress
  delaycompress
  notifempty
}

Only have to add the suitable filename. In short, those are the changes to be made:

  • Correct the crontab entry for gitlab-psql user:
0 0 * * * /opt/wal-e/bin/backup.sh >> /var/log/wal-e/wal-e_backup_push.log 2>&1
  • Add the wal-e backup filename in the logrotate.d/wal-e to suit the needs:
root@patroni-02-db-gprd.c.gitlab-production.internal:/etc/logrotate.d# cat wal-e 
/var/log/wal-e/wal-e.log /var/log/wal-e/wal-e_backup_push.log {
  daily
  missingok
  rotate 7
  compress
  delaycompress
  notifempty
}

The changes will apply immediately. No restart needed on any service.

Edited by Jose Finotto