Provide a supported mechanism to pause runtime DDL for logical-replication-based upgrades (e.g. AWS RDS Blue/Green deployments)

Problem to solve

GitLab performs DDL at runtime, outside of upgrade migrations, via scheduled Sidekiq cron workers. This breaks any workflow based on PostgreSQL logical replication, most notably AWS RDS Blue/Green deployments, which are the recommended low-downtime path for major PostgreSQL version upgrades on RDS.

When a Blue/Green deployment is created against a GitLab database, the green environment enters REPLICATION_DEGRADED and switchover is blocked:

Data definition language (DDL) changes aren't supported for blue/green deployments.
These changes aren't replicated from the blue environment to the green environment,
and switchover will be blocked. Your green databases now have a status of
REPLICATION_DEGRADED. Delete and recreate your blue/green deployment and avoid
future DDL changes.

PostgreSQL logs confirm the DDL is issued by GitLab itself at runtime:

WARNING:  command will not be replicated to the green instance: "ALTER TABLE"
WARNING:  command will not be replicated to the green instance: "CREATE TABLE"
WARNING:  command will not be replicated to the green instance: "ALTER TABLE"
ERROR:    relation "table_without_models" does not exist
STATEMENT: SELECT pg_get_serial_sequence('table_without_models', 'id')
...
WARNING:  command will not be replicated to the green instance: "DROP TABLE"

The table_without_models reference points at Gitlab::Database::Partitioning::TableWithoutModel (lib/gitlab/database/partitioning.rb), confirming the source is dynamic partition management.

Sources of runtime DDL

  • Database::PartitionManagementWorker (cron 21 */6 * * *) — CREATE TABLE / ALTER TABLE ... ATTACH PARTITION for dynamically partitioned tables
  • Database::DropDetachedPartitionsWorker (cron 20 3 * * *) — DROP TABLE for detached partitions
  • Database::MonitorLockedTablesWorker (cron 30 7 */3 * *) — trigger DDL on multi-database setups
  • Database reindexing (database_reindexing ops flag) — REINDEX / CREATE INDEX CONCURRENTLY

AWS explicitly documents this limitation for RDS for PostgreSQL Blue/Green deployments: https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/blue-green-deployments-considerations.html#blue-green-deployments-limitations-postgres-logical

Current state

  • There is no supported, documented way for self-managed administrators to temporarily suspend runtime DDL. The only workarounds are ad-hoc (disabling Sidekiq::Cron::Jobs from the Rails console, which does not survive Sidekiq restarts) or accepting a full-downtime in-place major upgrade.
  • GitLab's own infrastructure team faced the same constraint when upgrading GitLab.com's PostgreSQL via logical replication and paused DDL internally (see container-registry#1574 (closed)).
  • Related work exists but is scoped to Geo logical-replication secondaries (#596973 (closed), #600560, #600561, #600564), which acknowledges these workers as runtime DDL sources but does not help self-managed admins performing a database engine upgrade.

Proposal

  1. Provide a supported switch (for example an ops feature flag or maintenance-mode setting, similar to the existing database_reindexing ops flag) that suspends all runtime DDL: partition creation, detached partition dropping, table-locking trigger management, and reindexing. The setting should persist across Sidekiq restarts.
  2. Document the procedure for major PostgreSQL upgrades of external databases via logical replication (RDS Blue/Green and equivalents): enable the switch, create the Blue/Green deployment, switch over, disable the switch — and the constraints that apply during the window (no GitLab upgrades / gitlab:db:* tasks).

Impact

Without this, customers running GitLab against Amazon RDS (a documented and recommended external database option) cannot use the primary low-downtime major-upgrade mechanism their platform provides, and must take full downtime for every major PostgreSQL upgrade.