Set up a way to check that database monitoring is actually working
We've twice had alerts fail to actually come through. Changes to the Alert Manager configuration and templates or other pieces of the postgres_exporter->prometheus->alert mangaer->slack stack make our current alerting setup quite fragile.
I would like to set up a regular job which makes a database update which will trigger an alert to #database. This can be a dedicated alert or we can trigger one of the existing severity:info alerts.
For instance we could have a job which does ALTER SYSTEM to some innocuous parameter once a day. Or we could create an unused replication slot. Or we could create a table specifically for this and set up an alert which fires whenever the data in the table is updated and then update it once a day. (perhaps there's already such a table in the database?)
Alternately we could do something like:
DO $$ BEGIN raise log 'This is a test log'; END $$;
And then add a rule to gitlab-cookbooks/gitlab-mtail/files/default/mtail/postgres.mtail to watch for that message.