shell timeout when running `gitlab-ctl replicate-geo-database`
This command can take a very long time to run when we replicate a large database. It looks like we are hitting a timeout in the shell wrapper.
# gitlab-ctl replicate-geo-database --host=postgres-02.db.geo1.gitlab.com --slot-name=secondary_geo2 --sslmode=verify-ca --user=gitlab_repmgr
---------------------------------------------------------------
WARNING: Make sure this script is run from the secondary server
---------------------------------------------------------------
This script will disable your local PostgreSQL database, and start
a backup/restore from the primary node at 'postgres-02.db.geo1.gitlab.com'
*** You have 30 seconds to hit CTRL-C ***
* Stopping PostgreSQL and all GitLab services
Enter the password for gitlab_repmgr@postgres-02.db.geo1.gitlab.com:
* Checking for replication slot secondary_geo2
* Creating replication slot secondary_geo2
* Backing up postgresql.conf
* Moving old data directory to '/var/opt/gitlab/postgresql/data.1511883165'
* Starting base backup as the replicator user (gitlab_repmgr)
pg_basebackup: initiating base backup, waiting for checkpoint to complete
pg_basebackup: checkpoint completed
transaction log start point: 2F43/4B000028 on timeline 3
/opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/mixlib-shellout-2.3.2/lib/mixlib/shellout/unix.rb:126:in `run_command': Command timed out after 1800s: (Mixlib::ShellOut::CommandTimeout)
Command exceeded allowed execution time, process terminated
---- Begin output of PGPASSFILE=/var/opt/gitlab/postgresql/.pgpass /opt/gitlab/embedded/bin/pg_basebackup -h postgres-02.db.geo1.gitlab.com -p 5432 -D /var/opt/gitlab/postgresql/data -U gitlab_repmgr -v -x -P ----
STDOUT:
STDERR: pg_basebackup: initiating base backup, waiting for checkpoint to complete
pg_basebackup: checkpoint completed
transaction log start point: 2F43/4B000028 on timeline 3
39549505/955200070 kB (4%), 0/1 tablespace (...esql/data/base/16385/21971303.72)
---- End output of PGPASSFILE=/var/opt/gitlab/postgresql/.pgpass /opt/gitlab/embedded/bin/pg_basebackup -h postgres-02.db.geo1.gitlab.com -p 5432 -D /var/opt/gitlab/postgresql/data -U gitlab_repmgr -v -x -P ----
Ran PGPASSFILE=/var/opt/gitlab/postgresql/.pgpass /opt/gitlab/embedded/bin/pg_basebackup -h postgres-02.db.geo1.gitlab.com -p 5432 -D /var/opt/gitlab/postgresql/data -U gitlab_repmgr -v -x -P returned
from /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/mixlib-shellout-2.3.2/lib/mixlib/shellout.rb:263:in `run_command'
from /opt/gitlab/embedded/service/omnibus-ctl/lib/gitlab_ctl/util.rb:31:in `run_command'
from /opt/gitlab/embedded/service/omnibus-ctl-ee/lib/geo/replication.rb:151:in `run_command'
from /opt/gitlab/embedded/service/omnibus-ctl-ee/lib/geo/replication.rb:68:in `execute'
from /opt/gitlab/embedded/service/omnibus-ctl-ee/replicate_geo_database.rb:31:in `execute!'
from /opt/gitlab/embedded/service/omnibus-ctl-ee/replicate_geo_database.rb:5:in `block in load_file'
from /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/omnibus-ctl-0.5.0/lib/omnibus-ctl.rb:202:in `block in add_command_under_category'
from /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/omnibus-ctl-0.5.0/lib/omnibus-ctl.rb:730:in `run'
from /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/omnibus-ctl-0.5.0/bin/omnibus-ctl:31:in `<top (required)>'
from /opt/gitlab/embedded/bin/omnibus-ctl:23:in `load'
from /opt/gitlab/embedded/bin/omnibus-ctl:23:in `<main>'