## Summary At many occasions self-managed instances upgrade fail because of problems with their underlying database. There were below efforts done in past to bring any useful information related to their database inconsistencies to the forefront. * [Database Diagnostics](https://gitlab.com/gitlab-org/gitlab/-/work_items/567561) - Web UI * Schema inconsistencies * Collation corruption * [Database SOS](https://gitlab.com/groups/gitlab-com/-/work_items/2456) - Rake task (runs for 5 mins) * Samples every 3s: pg_stat_statements, table stats, pg_locks, pg_stat_user_tables, pg_stat_user_indexes * one off: Various Db settings, Collation check, BBM status In addition to these, there were escalations from customers because of not ideally configured database, patroni, pgbouncer and gitlab load-balancer. So we need an unified framework, which will allow admins to monitor their database state and ensure it's healthy enough for upgrades. ## Business Justification A big part of our upgrade related issues stem from databases in an inconsistent state. Customers need visibility into database health issues that could impact their GitLab instance (and also their ability to upgrade). The support team needs a unified way to detect and diagnose database inconsistencies. This will make self-managed instances to fix their database, so that their upgrade experience will be smoother and more predictable. Will also reduce the time spent on troubleshooting database related customer tickets. ## Expected impact - One framework to deduct and resolve (if possible) common database health problems - Being proactive in fixing db problems through alerts - Reduced support burden by surfacing issues earlier in the lifecycle ## Exit criteria * [ ] Admins able to see all database problems in one place * [ ] Classify problems based on the severity and their impact * [ ] Alerts set up to warn database problems * [ ] Documentation ## Deliverables - [ ] Phase 0 - Architecture - [ ] Architecture blueprint ready - [ ] Epic is broken down into smaller issues as needed - [ ] Project is marked ~"workflow::ready for development" - [ ] Phase 1 - State Metrics - [ ] Combine Database diagnostics and selected items from db:sos reports. - [ ] Identify and include configurable metrics (database, patroni, pgbouncer, LB settings) and their optimal values. - [ ] Classify all the above identified metrics with severity. Eg: Few metrics can be a show stopped but others are not. - [ ] Define overall health status of the database based on the identified metrics and their severity. - [ ] Phase 2 - Implementation - [ ] Web UI for admins to monitor all the state metrics - [ ] Design alerting for the database state metrics - [ ] Phase 3 - Documentation * [ ] Document about the framework and share it with stakeholders