Monitor locked tables

What does this MR do and why?

As part of the database decomposition project we made a copy of the database on a new cluster. We used the first database cluster, called main for all the database tables that belong to the gitlab_main gitlab schema. And used the new database cluster ci for all the tables that belong to the gitlab_schema gitlab_ci. While the gitlab_shared tables are still used on both clusters

After the decomposition we use gitlab_main and gitlab_shared tables on the database cluster main. And we use the gitlab_ci and gitlab_shared tables on the database cluster ci. But the database schema (not to be confused with gitlab_schema) is still the same on both database clusters. We are keeping it now like this for simplicity. That means we still have some legacy tables, that we have been truncated, that the application should not write to. These tables have been locked for writes, and we automatically lock writes on any newly created tables on the other database where the table is not used.

With this MR we are adding monitoring sidekiq job, that runs every 3 days and checks for any tables that for some reason haven't been locked.

Addressing: #396227 (closed)

How to set up and validate locally

Steps to validate locally

1. Make sure you are running in decomposed database setup
2. Lock the writes on the legacy tables rake gitlab:db:lock_writes
3. Enable the feature flag Feature.enable(:monitor_database_locked_tables)
4. Run the job the rails console: Database::MonitorLockedTablesWorker.perform_in(0).
5. Monitor the sidekiq jobs tail -f log/sidekiq.log. You should see the output of the job Database::MonitorLockedTablesWorker, with something like this "extra.database_monitor_locked_tables_worker.results":{"main":{"tables_need_lock":[],"tables_need_unlock":[],"tables_need_lock_count":0,"tables_need_unlock_count":0},"ci":{"tables_need_lock":["pm_advisories","pm_affected_packages","value_stream_dashboard_aggregations"],"tables_need_unlock":[],"tables_need_lock_count":3,"tables_need_unlock_count":0}}

Clean up

1. Unlock the writes on the tables if they weren't locked before rake gitlab:db:unlock_writes
2. (Extra test) Feel free to run the job again, and see many tables reported that need to be locked in the log of th eidekiq

MR acceptance checklist

This checklist encourages us to confirm any changes have been analyzed to reduce risks in quality, performance, reliability, security, and maintainability.

• I have evaluated the MR acceptance checklist for this MR.

Related to #396227 (closed)