Follow-up to patroni-embedding disks filling up on gstg

This is a follow-up issue for production#14886 (closed) (some more context there).

Tasks needed:

  • We have not solved the disk space consumption issue. We need to investigate and fix it. We have ~20 days before the disks fill up again.
  • We need to recover node 01 and make sure it becomes a replica again.
  • We need to fix node 02 so that it actually becomes a replica.
  • We should fix /var/opt/gitlab/postgresql/.pgpass to contain the correct password, this should also make gitlab-psql work on these boxes.
  • Postgres logs still contain a lot of authentication failures, we need to understand these.
Edited by Devin Sylva