Add partition_manager_sync_partitions ops flags per database
What does this MR do and why?
Partition management (sync_partitions, drop_detached_partitions and finalize_pending_detach_partitions in Gitlab::Database::Partitioning) is gated by a single ops flag, partition_manager_sync_partitions, for every database. During INC-15918 and INC-16023 that flag had to be disabled to stop repeated ATTACH PARTITION lock retries on the CI database, which also stopped partition retention on main and left detached web_hook_logs_daily partitions undropped there.
This MR adds one ops flag per database, partition_manager_sync_partitions_main, _ci and _sec, all enabled by default:
- Callers now name the database they operate on.
blocking_feature_flag(database_name)returns the global flag when it blocks, otherwise that database's flag when it is disabled. drop_detached_partitionsandfinalize_pending_detach_partitionstake adatabase_nameand run only on that database.Database::DropDetachedPartitionsWorkerandDatabase::FinalizePendingDetachPartitionsWorkeriterate the databases and call them per database.sync_partitionschecks the per-database flag once per database insync_database, and passesonly_onto the top-level check sogitlab:db:create_dynamic_partitions:<db>returns and prints the flag that blocked it.- The global
partition_manager_sync_partitionsflag keeps its behaviour as the kill switch for all databases.
With this, /chatops run feature set partition_manager_sync_partitions_ci false stops CI partition creation, detach and drop while main and sec keep running.
References
- Issue: https://gitlab.com/gitlab-org/gitlab/-/work_items/638350
- Change request that re-enabled the global flag: gitlab-com/gl-infra/production#23428 (closed)
- INC-15918: gitlab-com/gl-infra/production#23392
- INC-16023: https://app.incident.io/gitlab/incidents/16023 (no GitLab issue yet; the incident is still in triage)
- Change request deleting the preparing
ci_partitionsrow as a stopgap: gitlab-com/gl-infra/production#23436 (closed) - Runbook: https://gitlab.com/gitlab-com/runbooks/-/blob/master/docs/patroni-ci/ci-partition-attach-lock-contention.md
Screenshots or screen recordings
No UI change.
How to set up and validate locally
Requires a GDK with separate main and ci databases.
-
In a Rails console, disable the CI flag and run a sync:
Feature.disable(:partition_manager_sync_partitions_ci) Gitlab::Database::Partitioning.sync_partitionslog/application.logshowsSkipping sync of dynamic postgres partitionswith"feature_flag":"partition_manager_sync_partitions_ci"and"database_name":"ci", while themainsync runs. -
Run the per-database task for each database:
bin/rake gitlab:db:create_dynamic_partitions:ci bin/rake gitlab:db:create_dynamic_partitions:mainThe first prints
Note: partition_manager_sync_partitions_ci feature is currently disabled; the second prints nothing. -
Check the per-database retention methods:
Gitlab::Database::Partitioning.drop_detached_partitions('ci') # => :partition_manager_sync_partitions_ci Gitlab::Database::Partitioning.drop_detached_partitions('main') # runs the dropper on main -
Re-enable the flag with
Feature.enable(:partition_manager_sync_partitions_ci)and repeat step 1; both databases sync.
MR acceptance checklist
Evaluate this MR against the MR acceptance checklist. It helps you analyze changes to reduce risks in quality, performance, reliability, security, and maintainability.