Rake task gitlab:db:lock_writes should include table partitions
What does this MR do and why?
Addressing issue: #388308 (closed)
Background:
- Since we introduced the support for multiple databases which is the current setup on GitLab.com (production) and (staging), we have added a rake task to lock the writes on the
gitlab_maintables on the databaseci, andgitlab_citables on themaindatabase. With this MR, we are also including the attached partitions of the tables to into the locking mechanism. With this we also make sure that partitions are also not written to directly from the application. Because that's not supposed to happen anyway.
Some other small issues to address in this issue.
- Renaming
letblocks and changinginclude_examplestoit_behaves_like. Because using the sameletwith parameterized variables causes a caching problem in thelet. With this it returns the wrong variable. See the 2nd warning here. - Not using the same
instanceobject as the default one for tests. BecauseTablesLockercallslock_writesandunlock_writesin each run. But we need to test against a particular table only. Eitherlockorunlock.
How to set up and validate locally
Steps to validate locally
-
Make sure you are running in decomposed database mode
-
Create test partitions on both
mainandci
export PSQL_STATEMENT="CREATE TABLE gitlab_partitions_dynamic.test_partition PARTITION OF security_findings FOR VALUES IN (0)"
export MAIN_DATABASE=$(cat config/database.yml|yq .development.main.database)
export CI_DATABASE=$(cat config/database.yml|yq .development.ci.database)
gdk psql -d $MAIN_DATABASE -c $PSQL_STATEMENT
gdk psql -d $CI_DATABASE -c $PSQL_STATEMENT
- Run this rake task, to lock the writes on the
gitlab_citables on themain, andgitlab_mainon thecidatabase
VERBOSE=1 ./bin/rake gitlab:db:lock_writes
- Try to write/delete from the partitions on both
mainandci
PSQL_STATEMENT="DELETE FROM gitlab_partitions_dynamic.test_partition"
It should not fail on main
gdk psql -d $MAIN_DATABASE -c $PSQL_STATEMENT
But it should fail on ci
gdk psql -d $CI_DATABASE -c $PSQL_STATEMENT
- Clean up
- Unlock the writes on all the tables
VERBOSE=1 ./bin/rake gitlab:db:unlock_writes
- Drop the partitions
PSQL_STATEMENT="DROP TABLE gitlab_partitions_dynamic.test_partition"
gdk psql -d $MAIN_DATABASE -c $PSQL_STATEMENT
gdk psql -d $CI_DATABASE -c $PSQL_STATEMENT
Related Followup
In this related follow up issue: #391638 (closed) we are going to include tests in a similar fashion for the detached partitions
MR acceptance checklist
This checklist encourages us to confirm any changes have been analyzed to reduce risks in quality, performance, reliability, security, and maintainability.
-
I have evaluated the MR acceptance checklist for this MR.
Related to #388308 (closed)
Edited by Omar Qunsul