Geo: Backup failed during "Initiate the replication process" - `FATAL: password authentication failed for user "gitlab"`
Error while following https://docs.gitlab.com/ee/administration/geo/replication/database.html#step-3-initiate-the-replication-process:
root@mike-geo-instance-template-2:~# gitlab-ctl replicate-geo-database --slot-name=secondary_example --host=10.128.0.22
---------------------------------------------------------------
WARNING: Make sure this script is run from the secondary server
---------------------------------------------------------------
*** You are about to delete your local PostgreSQL database, and replicate the primary database. ***
*** The primary geo node is `10.128.0.22` ***
*** Are you sure you want to continue (replicate/no)? ***
Confirmation: replicate
* Executing GitLab backup task to prevent accidental data loss
Dumping database ...
Dumping PostgreSQL database gitlabhq_production ... [FAILED]
pg_dump: [archiver (db)] connection to database "gitlabhq_production" failed: FATAL: password authentication failed for user "gitlab"
FATAL: password authentication failed for user "gitlab"
Backup failed
*** Initial replication failed! ***
Replication tool returned with a non zero exit status!
Troubleshooting tips:
- replication should be run by root user
- check your trust settings `md5_auth_cidr_addresses` in `gitlab.rb` on the primary node
Failed to execute: gitlab-rake gitlab:backup:create
root@mike-geo-instance-template-2:~# vim /etc/gitlab/gitlab.rb
root@mike-geo-instance-template-2:~# gitlab-ctl replicate-geo-database --slot-name=secondary_example --host=10.128.0.22 --skip-backup
---------------------------------------------------------------
WARNING: Make sure this script is run from the secondary server
---------------------------------------------------------------
*** You are about to delete your local PostgreSQL database, and replicate the primary database. ***
*** The primary geo node is `10.128.0.22` ***
*** Are you sure you want to continue (replicate/no)? ***
Confirmation: replicate
* Stopping PostgreSQL and all GitLab services
Enter the password for gitlab_replicator@10.128.0.22:
* Checking for replication slot secondary_example
* Creating replication slot secondary_example
* Backing up postgresql.conf
* Moving old data directory to '/var/opt/gitlab/postgresql/data.1520958818'
* Starting base backup as the replicator user (gitlab_replicator)
pg_basebackup: initiating base backup, waiting for checkpoint to complete
pg_basebackup: checkpoint completed
transaction log start point: 0/2000060 on timeline 1
pg_basebackup: starting background WAL receiver
41206/41206 kB (100%), 1/1 tablespace
transaction log end point: 0/2000130
pg_basebackup: waiting for background process to finish streaming ...
pg_basebackup: base backup completed
* Writing recovery.conf file with sslmode=verify-ca
* Restoring postgresql.conf
* Setting ownership permissions in PostgreSQL data directory
* Starting PostgreSQL and all GitLab services
root@mike-geo-instance-template-2:~#Possible fixes
Backup is done by gitlab rake task, so it needs a fix there
cc @brodock