gitlab_pm/gitlab_main_clusterwide/gitlab_main_cell tables are not locked on the CI database
As part of the database decomposition project we made a copy of the database on a new cluster. We used the first database cluster, called main for all the database tables that belong to the gitlab_main gitlab schema. And used the new database cluster ci for all the tables that belong to the gitlab_schema gitlab_ci. While the gitlab_shared tables are still used on both clusters
After the decomposition we use gitlab_main and gitlab_shared tables on the database cluster main. And we use the gitlab_ci and gitlab_shared tables on the database cluster ci. But the database schema (not to be confused with gitlab_schema) is still the same on both database clusters. We are keeping it now like this for simplicity. That means we still have some legacy tables, that we have been truncated, that the application should not write to. These tables have been locked for writes, and we automatically lock writes on any newly created tables on the other database where the table is not used.
For some reason, the automatic locking of writes on any newly created tables doesn't include gitlab_pm tables. Which at the moment are located on the main database. But they should be locked on the ci database in a decomposed database setup.
As part of this issue:
- Make sure that
rake gitlab:db:lock_writeslocksgitlab_pm/gitlab_main_clusterwide/gitlab_main_celltables on thecidatabase. Maybe that's already the case, but verify. - Confirmed to be broken, make sure that any newly created
gitlab_pmtable is locked on thecidatabase, and kept unlocked on themaindatabase.. This is done by fixing the code in the file https://gitlab.com/gitlab-org/gitlab/-/blob/master/lib/gitlab/database/migration_helpers/automatic_lock_writes_on_tables.rb
CC: @tkuah @rutgerwessels