Monitor table lock writes status on GitLab.com

Problem

We have CI decomposed databases on GitLab.com. To ensure integrity we lock-writes tables that are not supposed to be written to (ci tables on main DB, and vice versa)

We have run gitlab:db:lock_writes a few times already. Now we should have monitoring to ensure all tables have correct lock-writes status.

Proposal

Ideas:

• What about adding this to the database checkup reports ? This is the script running: https://gitlab.com/postgres-ai/postgres-checkup/
• Add lock write status to Prometheus. perhaps using a database probe in gitlab-exporter (https://gitlab.com/gitlab-org/gitlab-exporter/#supported-probes)? If we have this in metrics, it means we can fire an alert and notify a slack channel that the task needs to be run again?

/cc @OmarQunsulGitlab @rutgerwessels @rhenchen.gitlab @DylanGriffith @alexives