The `gitlab-psql` crontab is misconfigured on Patroni backup replica nodes
Related incidents:
- https://gitlab.com/gitlab-com/gl-infra/production/-/issues/5759
- https://gitlab.com/gitlab-com/gl-infra/production/-/issues/5762
It was initially assumed in #14444 (closed) that the cron job gcs-snapshot.sh was supposed to be in the crontab for gitlab-psql, which was correct on 16.04 nodes until their upgrade to 20.04 where the cron job moved to the root crontab.
So the root crontab is correct:
pguinoiseau@patroni-v12-registry-03-db-gprd.c.gitlab-production.internal:~$ sudo crontab -l
# Chef Name: GCS snapshot
PATH="/usr/local/sbin:/usr/sbin/:/sbin:/usr/local/bin:/usr/bin:/bin:/snap/bin"
0 */6 * * * /usr/local/bin/gcs-snapshot.sh
And the cron job entry should be removed from the gitlab-psql crontab:
pguinoiseau@patroni-v12-registry-03-db-gprd.c.gitlab-production.internal:~$ sudo crontab -l -u gitlab-psql
[...]
# Chef Name: GCS snapshot
PATH="/usr/local/sbin:/usr/sbin/:/sbin:/usr/local/bin:/usr/bin:/bin"
0 */6 * * * /usr/local/bin/gcs-snapshot.sh
[...]