Transient failure in `ee:geo`
I think the problem is timeout: run: geo-logcursor: (pid 2559) 58s, want down, got TERM below:
8560 *** You are about to delete your local PostgreSQL database, and replicate the primary database. ***
8561 *** The primary geo node is `gitlab-primary.geo` ***
8562 * Stopping PostgreSQL and all GitLab services
8563 ok: down: alertmanager: 0s, normally up
8564 timeout: run: geo-logcursor: (pid 2559) 58s, want down, got TERM
8565 ok: down: geo-postgresql: 0s, normally up
8566 ok: down: gitaly: 1s, normally up
8567 ok: down: gitlab-exporter: 0s, normally up
8568 ok: down: gitlab-workhorse: 1s, normally up
8569 ok: down: grafana: 0s, normally up
8570 ok: down: logrotate: 0s, normally up
8571 ok: down: nginx: 1s, normally up
8572 ok: down: postgres-exporter: 0s, normally up
8573 ok: down: postgresql: 1s, normally up
8574 ok: down: prometheus: 0s, normally up
8575 ok: down: redis: 1s, normally up
8576 ok: down: redis-exporter: 0s, normally up
8577 ok: down: sidekiq: 0s, normally up
8578 ok: down: sshd: 0s, normally up
8579 ok: down: unicorn: 1s, normally up
8580 *** Initial replication failed! ***
8581 Replication tool returned with a non zero exit status!
8582 Troubleshooting tips:
8583 - replication should be run by root user
8584 - check if `roles ['geo_primary_role']` or `geo_primary_role['enable'] = true` exists in `gitlab.rb` on the primary node
8585 - check your trust settings `md5_auth_cidr_addresses` in `gitlab.rb` on the primary node
8586 Failed to execute: gitlab-ctl stop
8587 bundler: failed to load command: bin/qa (bin/qa)
8588 QA::Service::Shellout::CommandError: Command `docker exec gitlab-secondary bash -c 'echo mypass | gitlab-ctl replicate-geo-database --host=gitlab-primary.geo --slot-name=gitlab_primary --sslmode=disable --no-wait --force'` failed!
8589 /home/gitlab/qa/qa/service/shellout.rb:25:in `block in shell'
8590 /usr/local/lib/ruby/2.6.0/open3.rb:219:in `popen_run'
8591 /usr/local/lib/ruby/2.6.0/open3.rb:208:in `popen2e'
8592 /home/gitlab/qa/qa/service/shellout.rb:19:in `shell'
8593 /home/gitlab/qa/qa/service/omnibus.rb:19:in `docker_exec'
8594 /home/gitlab/qa/qa/service/omnibus.rb:14:in `gitlab_ctl'
8595 /home/gitlab/qa/qa/ee/scenario/test/geo.rb:122:in `block in replicate_database'
8596 /home/gitlab/qa/qa/scenario/actable.rb:7:in `instance_exec'
8597 /home/gitlab/qa/qa/scenario/actable.rb:7:in `act'
8598 /home/gitlab/qa/qa/ee/scenario/test/geo.rb:116:in `replicate_database'
8599 /home/gitlab/qa/qa/ee/scenario/test/geo.rb:37:in `block in perform'
8600 /home/gitlab/qa/qa/scenario/actable.rb:7:in `instance_exec'
8601 /home/gitlab/qa/qa/scenario/actable.rb:7:in `act'
8602 /home/gitlab/qa/qa/scenario/actable.rb:20:in `act'
8603 /home/gitlab/qa/qa/ee/scenario/test/geo.rb:36:in `perform'
8604 /home/gitlab/qa/qa/scenario/template.rb:10:in `block in perform'
8605 /home/gitlab/qa/qa/scenario/template.rb:8:in `tap'
8606 /home/gitlab/qa/qa/scenario/template.rb:8:in `perform'
8607 /home/gitlab/qa/qa/scenario/bootable.rb:28:in `launch!'
8608 bin/qa:7:in `<top (required)>'
8609 Docker shell command: `docker stop gitlab-primary`
8610 gitlab-primary
8611 Docker shell command: `docker rm -f gitlab-primary`
8612 /builds/gitlab-org/gitlab-qa/lib/gitlab/qa/docker/shellout.rb:31:in `block in execute!': Docker command `docker run -t --rm --net=geo --env GITHUB_ACCESS_TOKEN="$GITHUB_ACCESS_TOKEN" --env EE_LICENSE="$EE_LICENSE" --env GCLOUD_ACCOUNT_EMAIL="$GCLOUD_ACCOUNT_EMAIL" --env GCLOUD_ACCOUNT_KEY="$GCLOUD_ACCOUNT_KEY" --env CLOUDSDK_CORE_PROJECT="$CLOUDSDK_CORE_PROJECT" --env GCLOUD_REGION="$GCLOUD_REGION" --env QA_DEBUG="$QA_DEBUG" --env QA_CAN_TEST_GIT_PROTOCOL_V2="$QA_CAN_TEST_GIT_PROTOCOL_V2" --env GITHUB_OAUTH_APP_ID="$GITHUB_OAUTH_APP_ID" --env GITHUB_OAUTH_APP_SECRET="$GITHUB_OAUTH_APP_SECRET" --env GITHUB_USERNAME="$GITHUB_USERNAME" --env GITHUB_PASSWORD="$GITHUB_PASSWORD" --env CI="$CI" --env CI_RUNNER_ID="$CI_RUNNER_ID" --env CI_SERVER_HOST="$CI_SERVER_HOST" --env CI_NODE_TOTAL="$CI_NODE_TOTAL" --env GITLAB_CI="$GITLAB_CI" --env GITLAB_QA_1P_EMAIL="$GITLAB_QA_1P_EMAIL" --env GITLAB_QA_1P_PASSWORD="$GITLAB_QA_1P_PASSWORD" --env GITLAB_QA_1P_SECRET="$GITLAB_QA_1P_SECRET" --env GITLAB_QA_1P_GITHUB_UUID="$GITLAB_QA_1P_GITHUB_UUID" --volume /var/run/docker.sock:/var/run/docker.sock:z --volume /builds/gitlab-org/gitlab-qa/gitlab-qa-run-2020-01-09-06-40-29-4ed21de3/gitlab-ee-qa-9e801808:/home/gitlab/qa/tmp:z --name gitlab-ee-qa-9e801808 dev.gitlab.org:5005/gitlab/omnibus-gitlab/gitlab-ee-qa:12.7-e5c967cc40e QA::EE::Scenario::Test::Geo --primary-address http://gitlab-primary.geo --primary-name gitlab-primary --secondary-address http://gitlab-secondary.geo --secondary-name gitlab-secondary -- --format RspecJunitFormatter --out "tmp/rspec-832708.xml" --format html --out "tmp/rspec-$(echo $CI_JOB_NAME | sed 's/[ /]/_/g').htm" --color --format documentation` failed! (Gitlab::QA::Docker::Shellout::StatusError)
8613 from /usr/local/lib/ruby/2.4.0/open3.rb:205:in `popen_run'
8614 from /usr/local/lib/ruby/2.4.0/open3.rb:190:in `popen2e'
8615 from /builds/gitlab-org/gitlab-qa/lib/gitlab/qa/docker/shellout.rb:19:in `execute!'
8616 from /builds/gitlab-org/gitlab-qa/lib/gitlab/qa/docker/command.rb:36:in `execute!'
8617 from /builds/gitlab-org/gitlab-qa/lib/gitlab/qa/docker/engine.rb:22:in `block in run'
8618 from /builds/gitlab-org/gitlab-qa/lib/gitlab/qa/docker/engine.rb:16:in `tap'
8619 from /builds/gitlab-org/gitlab-qa/lib/gitlab/qa/docker/engine.rb:16:in `run'
8620 from /builds/gitlab-org/gitlab-qa/lib/gitlab/qa/component/specs.rb:36:in `perform'
8621 from /builds/gitlab-org/gitlab-qa/lib/gitlab/qa/scenario/template.rb:8:in `block in perform'
8622 from /builds/gitlab-org/gitlab-qa/lib/gitlab/qa/scenario/template.rb:6:in `tap'
8623 from /builds/gitlab-org/gitlab-qa/lib/gitlab/qa/scenario/template.rb:6:in `perform'
8624 from /builds/gitlab-org/gitlab-qa/lib/gitlab/qa/scenario/test/integration/geo.rb:69:in `block (4 levels) in perform'
8625 from /builds/gitlab-org/gitlab-qa/lib/gitlab/qa/scenario/actable.rb:6:in `instance_exec'
8626 from /builds/gitlab-org/gitlab-qa/lib/gitlab/qa/scenario/actable.rb:6:in `act'
8627 from /builds/gitlab-org/gitlab-qa/lib/gitlab/qa/scenario/test/integration/geo.rb:59:in `block (3 levels) in perform'
8628 from /builds/gitlab-org/gitlab-qa/lib/gitlab/qa/scenario/actable.rb:15:in `perform'
8629 from /builds/gitlab-org/gitlab-qa/lib/gitlab/qa/scenario/test/integration/geo.rb:40:in `block (2 levels) in perform'
8630 from /builds/gitlab-org/gitlab-qa/lib/gitlab/qa/component/gitlab.rb:83:in `instance'
8631 from /builds/gitlab-org/gitlab-qa/lib/gitlab/qa/scenario/test/integration/geo.rb:39:in `block in perform'
8632 from /builds/gitlab-org/gitlab-qa/lib/gitlab/qa/scenario/actable.rb:15:in `perform'
8633 from /builds/gitlab-org/gitlab-qa/lib/gitlab/qa/scenario/test/integration/geo.rb:19:in `perform'
8634 from /builds/gitlab-org/gitlab-qa/lib/gitlab/qa/scenario/template.rb:8:in `block in perform'
8635 from /builds/gitlab-org/gitlab-qa/lib/gitlab/qa/scenario/template.rb:6:in `tap'
8636 from /builds/gitlab-org/gitlab-qa/lib/gitlab/qa/scenario/template.rb:6:in `perform'
8637 from /builds/gitlab-org/gitlab-qa/lib/gitlab/qa/runner.rb:42:in `run'
8638 from exe/gitlab-qa:8:in `<main>'
8639 gitlab-primary