Monitor table lock writes status on GitLab.com
Problem
We have CI decomposed databases on GitLab.com. To ensure integrity we lock-writes tables that are not supposed to be written to (ci tables on main DB, and vice versa)
We have run gitlab:db:lock_writes
a few times already. Now we should have monitoring to ensure all tables have correct lock-writes status.
Proposal
Ideas:
- What about adding this to the database checkup reports ? This is the script running: https://gitlab.com/postgres-ai/postgres-checkup/
- Add lock write status to Prometheus. perhaps using a database probe in gitlab-exporter (https://gitlab.com/gitlab-org/gitlab-exporter/#supported-probes)? If we have this in metrics, it means we can fire an alert and notify a slack channel that the task needs to be run again?
/cc @OmarQunsulGitlab @rutgerwessels @rhenchen.gitlab @DylanGriffith @alexives
Edited by Thong Kuah