Newly provisioned postgres fleet requires reboot for clean /var/log mount

Background

We recently provisioned new postgres clusters on 20.04, and switched to those over the weekend: https://ops.gitlab.net/gitlab-com/gl-infra/db-migration/-/issues/43.

Unfortunately we ran into the same pathology that we did during the gitaly OS upgrade: scalability#1418.

Problem

During the initial boot, the bootstrap script detects the log disk and mounts it. But this happens too late. The OS has already opened file descriptors on the root filesystem's /var/log.

Thus, /var/log/syslog is no longer accessible, and so is other stuff that ordinarily would be, for example:

iwiedler@patroni-main-2004-10-db-gprd.c.gitlab-production.internal:~$ sudo last

last: cannot open /var/log/wtmp: No such file or directory

This poses a stability risk, and limits our ability to respond to issues, as logs are hidden away.

Proposal

We should reboot the entire postgres fleet, so that we have a clean mount early on in the boot process (via fstab).