Sync sequences before Geo promotion with logical replication

What does this MR do and why?

PostgreSQL logical replication copies table rows but never sequence values (this is a documented restriction). After promoting a Geo secondary (gitlab-ctl geo promote, which calls rake geo:set_secondary_as_primary), every sequence is still at its start value. The first INSERT on the new primary reuses an existing ID and fails with PG::UniqueViolation / ActiveRecord::RecordNotUnique.

This MR adds Gitlab::Database::SyncSequencesWithTableData (CE) and wires it into geo:set_secondary_as_primary (EE) when Gitlab::Geo::HealthCheck#logical_replication_mode? is true, gated by the WIP flag geo_postgresql_replication_agnostic. It runs one catalog query per database (main, ci, sec) to find every sequence and the tables it feeds, resolves partitioned tables to their root parent, takes MAX(col) across all consumers, adds a 1000-ID buffer, and advances the sequence with setval. It respects MINVALUE/START floors and MAXVALUE ceilings, refuses to run if any logical replication subscription still exists, and aborts the promotion (raising SyncError) if anything fails, before any GeoNode change is made.

This supersedes !231508 (closed). That MR discovered sequences two ways: sequences with an OWNED BY table were synced only against that one table, and un-owned sequences were synced against their nextval-default consumers. The schema has since drifted in ways that break this split: audit_events_id_seq is now OWNED BY NONE but feeds 5 tables, and the whole p_ci_* family has no nextval default at all (IDs are trigger-assigned), so ownership is the only catalog trace left for them. !231508 (closed)'s shared-sequence query also returned unqualified partition table names, which failed on real installations. Rebuilding discovery as a single query that unions both signals (and resolves partitions correctly) was cleaner than patching around this.

Database review

The full discovery query is SEQUENCES_WITH_CONSUMERS_SQL in lib/gitlab/database/sync_sequences_with_table_data.rb. It's catalog-only (no table scans): it unions pg_attrdef (nextval column defaults) with pg_depend (OWNED BY links, deptypes a/i), resolves partition children to their root parent via a recursive pg_inherits CTE, schema-qualifies every name, and groups consumers per sequence. On GDK with a full schema this takes about 190ms and finds 898 sequences and 940 consumer pairs.

For each sequence, we run MAX(col) per consumer table. This is index-backed for the well-indexed majority (index-only scans, including Merge Append over partitioned parents like p_ci_builds and uploads). About 9 sequences have consumers with no leading-column index (loose_foreign_keys_deleted_records, the five virtual_registries_* cache tables, value_stream_dashboard_counts, background_operation_*_cell_local) and seq-scan instead. We're treating that as an acceptable one-time cost during failover downtime, with statement_timeout disabled for the run.

TODO: attach postgres.ai plans for the discovery query and a representative MAX query in a comment.

Note for pipeline reviewers: check-ci-partition-pruning may warn on the SELECT MAX(id) FROM p_ci_* queries here. This is expected: they run once during failover downtime, not in the request path.

Also adds a standalone rake task, gitlab:geo:logical_replication:sync_sequences (optionally scoped with ONLY_SEQUENCES=name1,name2), so operators can repair sequences after the initial replication copy or re-run the sync without a promotion.

How to set up and validate locally

  1. Enable the feature flag: Feature.enable(:geo_postgresql_replication_agnostic).
  2. Lower a sequence with setval (below the actual max row value).
  3. In a rails console, run Gitlab::Database::SyncSequencesWithTableData.new.execute.
  4. Confirm the sequence advanced to MAX(col) + 1000.
  5. Optionally, validate end to end with a full two-node Geo LR setup per #593337 (closed).

Related to #593337 (closed). Supersedes !231508 (closed).

Edited by Douglas Barbosa Alexandre

Merge request reports

Loading
Loading