Skip to content

Cannot get transaction log after `gitlab-ctl replicate-geo-database `

After initiating replication on the Geo testbed (copy of the gitlab.com database) we ended with an error that indicates that xlog segments were unavailable after the data transfer. (see the error in https://gitlab.com/gitlab-com/infrastructure/issues/3481#note_53141868)

This was reported as well in https://gitlab.com/gitlab-org/gitlab-ee/issues/3463 where we bumped the default wal_keep_segments.

On the primary database that was being replicated we have the following settings:

# - Replication
wal_keep_segments = 512

max_standby_archive_delay = 30s # max delay before canceling queries
          # when reading WAL from archive;
          # -1 allows indefinite delay
max_standby_streaming_delay = 30s # max delay before canceling queries
          # when reading streaming WAL;
          # -1 allows indefinite delay

hot_standby_feedback = off   # send info from standby to prevent
          # query conflicts

I was able to complete replication after changing -x (same as -X fetch) to -X stream in the call to pg_base_backup https://gitlab.com/gitlab-org/omnibus-gitlab/blob/master/files/gitlab-ctl-commands-ee/lib/geo/replication.rb#L76

This issue is for discussing whether we want to update the omnibus configuration or the backup invocation.

/cc @_stark @stanhu