Geo failures when reconfiguring secondary: ruby_block[authorize Grafana with GitLab] fails with Command time out after 600s
Summary
In ee:geo test jobs on master for the GitLab project (for example https://gitlab.com/gitlab-org/gitlab-qa-mirror/-/jobs/814820730), Geo setup is failing during reconfigure of the secondary site (after running replicate-geo-database command) with error messages
QA::Service::Shellout::CommandError: Command `docker exec gitlab-secondary bash -c 'gitlab-ctl reconfigure'` failed! ==============================================================================
Error executing action `run` on resource 'ruby_block[authorize Grafana with GitLab]'
================================================================================
Mixlib::ShellOut::CommandTimeout
--------------------------------
Command timed out after 600s:
Command exceeded allowed execution time, process terminated
---- Begin output of /opt/gitlab/bin/gitlab-rails runner -e production 'app = Doorkeeper::Application.where(redirect_uri: "http://gitlab-secondary.geo/-/grafana/login/gitlab", name: "GitLab Grafana").first_or_create;puts app.uid.concat(" ").concat(app.secret);' ----The commands refer to the secondary Geo site, but it seems we shouldn't be running this on the secondary site. Is there a skip missing?
The earliest example of this happening that I've found is on October 15 (Pacific Time): https://gitlab.com/gitlab-org/gitlab-qa-mirror/-/jobs/793913115
Steps to reproduce
Run package-and-qa job on GitLab master (specifically the ee:geo test job)
Relevant logs
Example from: https://gitlab.com/gitlab-org/gitlab-qa-mirror/-/jobs/814820730
Recipe: monitoring::grafana
* directory[/var/log/gitlab/grafana] action create (up to date)
* directory[/var/opt/gitlab/grafana] action create (up to date)
* directory[/var/opt/gitlab/grafana/provisioning] action create (up to date)
* directory[/var/opt/gitlab/grafana/provisioning/dashboards] action create (up to date)
* directory[/var/opt/gitlab/grafana/provisioning/datasources] action create (up to date)
* directory[/var/opt/gitlab/grafana/provisioning/notifiers] action create (up to date)
* file[/var/opt/gitlab/grafana/CVE_reset_status] action delete (up to date)
* link[/var/opt/gitlab/grafana/conf] action create (up to date)
* link[/var/opt/gitlab/grafana/public] action create (up to date)
* directory[/opt/gitlab/etc/grafana/env] action create (up to date)
* ruby_block[authorize Grafana with GitLab] action runThere was an error running gitlab-ctl reconfigure:
ruby_block[authorize Grafana with GitLab] (monitoring::grafana line 95) had an error: Mixlib::ShellOut::CommandTimeout: Command timed out after 600s:
Command exceeded allowed execution time, process terminated
---- Begin output of /opt/gitlab/bin/gitlab-rails runner -e production 'app = Doorkeeper::Application.where(redirect_uri: "http://gitlab-secondary.geo/-/grafana/login/gitlab", name: "GitLab Grafana").first_or_create;puts app.uid.concat(" ").concat(app.secret);' ----
STDOUT:
STDERR:
---- End output of /opt/gitlab/bin/gitlab-rails runner -e production 'app = Doorkeeper::Application.where(redirect_uri: "http://gitlab-secondary.geo/-/grafana/login/gitlab", name: "GitLab Grafana").first_or_create;puts app.uid.concat(" ").concat(app.secret);' ----
Ran /opt/gitlab/bin/gitlab-rails runner -e production 'app = Doorkeeper::Application.where(redirect_uri: "http://gitlab-secondary.geo/-/grafana/login/gitlab", name: "GitLab Grafana").first_or_create;puts app.uid.concat(" ").concat(app.secret);' returned
================================================================================
Error executing action `run` on resource 'ruby_block[authorize Grafana with GitLab]'
================================================================================
Mixlib::ShellOut::CommandTimeout
--------------------------------
Command timed out after 600s:
Command exceeded allowed execution time, process terminated
---- Begin output of /opt/gitlab/bin/gitlab-rails runner -e production 'app = Doorkeeper::Application.where(redirect_uri: "http://gitlab-secondary.geo/-/grafana/login/gitlab", name: "GitLab Grafana").first_or_create;puts app.uid.concat(" ").concat(app.secret);' ----
STDOUT:
STDERR:
---- End output of /opt/gitlab/bin/gitlab-rails runner -e production 'app = Doorkeeper::Application.where(redirect_uri: "http://gitlab-secondary.geo/-/grafana/login/gitlab", name: "GitLab Grafana").first_or_create;puts app.uid.concat(" ").concat(app.secret);' ----
Ran /opt/gitlab/bin/gitlab-rails runner -e production 'app = Doorkeeper::Application.where(redirect_uri: "http://gitlab-secondary.geo/-/grafana/login/gitlab", name: "GitLab Grafana").first_or_create;puts app.uid.concat(" ").concat(app.secret);' returned
Cookbook Trace:
---------------
/opt/gitlab/embedded/cookbooks/cache/cookbooks/package/libraries/helpers/shell_out_helper.rb:6:in `do_shell_out'
/opt/gitlab/embedded/cookbooks/cache/cookbooks/gitlab/libraries/helpers/authorizer_helper.rb:6:in `query_gitlab_rails'
/opt/gitlab/embedded/cookbooks/cache/cookbooks/gitlab/libraries/helpers/grafana_helper.rb:12:in `authorize_with_gitlab'
/opt/gitlab/embedded/cookbooks/cache/cookbooks/monitoring/recipes/grafana.rb:97:in `block (2 levels) in from_file'
Resource Declaration:
---------------------
# In /opt/gitlab/embedded/cookbooks/cache/cookbooks/monitoring/recipes/grafana.rb
95: ruby_block "authorize Grafana with GitLab" do
96: block do
97: GrafanaHelper.authorize_with_gitlab(external_url)
98: end
99: # Try connecting to GitLab only if it is enabled and on this node
100: only_if { node['gitlab']['gitlab-rails']['enable'] }
101: end
102: end
Compiled Resource:
------------------
# Declared in /opt/gitlab/embedded/cookbooks/cache/cookbooks/monitoring/recipes/grafana.rb:95:in `from_file'
ruby_block("authorize Grafana with GitLab") do
action [:run]
default_guard_interpreter :default
declared_type :ruby_block
cookbook_name "monitoring"
recipe_name "grafana"
block #<Proc:0x0000000005534968 /opt/gitlab/embedded/cookbooks/cache/cookbooks/monitoring/recipes/grafana.rb:96>
block_name "authorize Grafana with GitLab"
only_if { #code block }
end
System Info:
------------
chef_version=15.14.0
platform=ubuntu
platform_version=16.04
ruby=ruby 2.7.2p137 (2020-10-01 revision 5445e04352) [x86_64-linux]
program_name=/opt/gitlab/embedded/bin/chef-client
executable=/opt/gitlab/embedded/bin/chef-client
Running handlers:
Deprecations:
* geo_secondary['db_fdw'] has been deprecated since 13.3 and will be removed in 14.0. Geo does not require Foreign Data Wrapper (FDW) to be configured to replicate data.
Running handlers complete
Chef Infra Client failed. 9 resources updated in 11 minutes 49 seconds
Deprecations:
* geo_secondary['db_fdw'] has been deprecated since 13.3 and will be removed in 14.0. Geo does not require Foreign Data Wrapper (FDW) to be configured to replicate data.
bundler: failed to load command: bin/qa (bin/qa)
QA::Service::Shellout::CommandError: Command `docker exec gitlab-secondary bash -c 'gitlab-ctl reconfigure'` failed!
/home/gitlab/qa/qa/service/shellout.rb:32:in `block in shell'
/usr/local/lib/ruby/2.7.0/open3.rb:219:in `popen_run'
/usr/local/lib/ruby/2.7.0/open3.rb:208:in `popen2e'
/home/gitlab/qa/qa/service/shellout.rb:19:in `shell'
/home/gitlab/qa/qa/service/omnibus.rb:19:in `docker_exec'
/home/gitlab/qa/qa/service/omnibus.rb:14:in `gitlab_ctl'
/home/gitlab/qa/qa/ee/scenario/test/geo.rb:140:in `block in reconfigure'
/home/gitlab/qa/qa/scenario/actable.rb:7:in `instance_exec'
/home/gitlab/qa/qa/scenario/actable.rb:7:in `act'
/home/gitlab/qa/qa/ee/scenario/test/geo.rb:139:in `reconfigure'
/home/gitlab/qa/qa/ee/scenario/test/geo.rb:38:in `block in perform'
/home/gitlab/qa/qa/scenario/actable.rb:7:in `instance_exec'
/home/gitlab/qa/qa/scenario/actable.rb:7:in `act'
/home/gitlab/qa/qa/scenario/actable.rb:20:in `act'
/home/gitlab/qa/qa/ee/scenario/test/geo.rb:36:in `perform'
/home/gitlab/qa/qa/scenario/template.rb:10:in `block in perform'
/home/gitlab/qa/qa/scenario/template.rb:8:in `tap'
/home/gitlab/qa/qa/scenario/template.rb:8:in `perform'
/home/gitlab/qa/qa/scenario/bootable.rb:28:in `launch!'
bin/qa:7:in `<top (required)>'
Docker shell command: `docker ps -f name=gitlab-primary`
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
0dfdc1fa4035 registry.gitlab.com/gitlab-org/build/omnibus-gitlab-mirror/gitlab-ee:ca27d956a7175c798fe688071d5a41d3e0faa719 "/assets/wrapper" 32 minutes ago Up 32 minutes (healthy) 22/tcp, 443/tcp, 0.0.0.0:32768->80/tcp gitlab-primary
Docker shell command: `docker stop gitlab-primary`
gitlab-primary
Docker shell command: `docker rm -f gitlab-primary`
/builds/gitlab-org/gitlab-qa-mirror/lib/gitlab/qa/docker/shellout.rb:31:in `block in execute!': Docker command `docker run -t --rm --net=geo --env GITHUB_ACCESS_TOKEN="$GITHUB_ACCESS_TOKEN" --env EE_LICENSE="$EE_LICENSE" --env GCLOUD_ACCOUNT_EMAIL="$GCLOUD_ACCOUNT_EMAIL" --env GCLOUD_ACCOUNT_KEY="$GCLOUD_ACCOUNT_KEY" --env CLOUDSDK_CORE_PROJECT="$CLOUDSDK_CORE_PROJECT" --env QA_DEBUG="$QA_DEBUG" --env QA_CAN_TEST_GIT_PROTOCOL_V2="$QA_CAN_TEST_GIT_PROTOCOL_V2" --env QA_CAN_TEST_PRAEFECT="$QA_CAN_TEST_PRAEFECT" --env GITHUB_USERNAME="$GITHUB_USERNAME" --env GITHUB_PASSWORD="$GITHUB_PASSWORD" --env CI="$CI" --env CI_JOB_ID="$CI_JOB_ID" --env CI_JOB_URL="$CI_JOB_URL" --env CI_RUNNER_ID="$CI_RUNNER_ID" --env CI_SERVER_HOST="$CI_SERVER_HOST" --env CI_SERVER_PERSONAL_ACCESS_TOKEN="$CI_SERVER_PERSONAL_ACCESS_TOKEN" --env CI_NODE_TOTAL="$CI_NODE_TOTAL" --env GITLAB_CI="$GITLAB_CI" --env CI_SLACK_WEBHOOK_URL="$CI_SLACK_WEBHOOK_URL" --env GITLAB_QA_FORMLESS_LOGIN_TOKEN="$GITLAB_QA_FORMLESS_LOGIN_TOKEN" --env JIRA_ADMIN_USERNAME="$JIRA_ADMIN_USERNAME" --env JIRA_ADMIN_PASSWORD="$JIRA_ADMIN_PASSWORD" --volume /var/run/docker.sock:/var/run/docker.sock:z --volume /builds/gitlab-org/gitlab-qa-mirror/gitlab-qa-run-2020-10-27-15-52-11-f3f119fb/gitlab-ee-qa-04670869:/home/gitlab/qa/tmp:z --name gitlab-ee-qa-04670869 registry.gitlab.com/gitlab-org/build/omnibus-gitlab-mirror/gitlab-ee-qa:ca27d956a7175c798fe688071d5a41d3e0faa719 QA::EE::Scenario::Test::Geo --primary-address http://gitlab-primary.geo --primary-name gitlab-primary --secondary-address http://gitlab-secondary.geo --secondary-name gitlab-secondary -- --format QA::Support::JsonFormatter --out "tmp/rspec-814820730.json" --format RspecJunitFormatter --out "tmp/rspec-814820730.xml" --format html --out "tmp/rspec-$(echo $CI_JOB_NAME | sed 's/[ /]/_/g').htm" --color --format documentation` failed! (Gitlab::QA::Docker::Shellout::StatusError)
from /usr/local/lib/ruby/2.6.0/open3.rb:219:in `popen_run'
from /usr/local/lib/ruby/2.6.0/open3.rb:208:in `popen2e'
from /builds/gitlab-org/gitlab-qa-mirror/lib/gitlab/qa/docker/shellout.rb:19:in `execute!'
from /builds/gitlab-org/gitlab-qa-mirror/lib/gitlab/qa/docker/command.rb:48:in `execute!'
from /builds/gitlab-org/gitlab-qa-mirror/lib/gitlab/qa/docker/engine.rb:27:in `block in run'
from /builds/gitlab-org/gitlab-qa-mirror/lib/gitlab/qa/docker/engine.rb:21:in `tap'
from /builds/gitlab-org/gitlab-qa-mirror/lib/gitlab/qa/docker/engine.rb:21:in `run'
from /builds/gitlab-org/gitlab-qa-mirror/lib/gitlab/qa/component/specs.rb:46:in `block in perform'
from /builds/gitlab-org/gitlab-qa-mirror/lib/gitlab/qa/component/specs.rb:45:in `each'
from /builds/gitlab-org/gitlab-qa-mirror/lib/gitlab/qa/component/specs.rb:45:in `perform'
from /builds/gitlab-org/gitlab-qa-mirror/lib/gitlab/qa/scenario/template.rb:8:in `block in perform'
from /builds/gitlab-org/gitlab-qa-mirror/lib/gitlab/qa/scenario/template.rb:6:in `tap'
from /builds/gitlab-org/gitlab-qa-mirror/lib/gitlab/qa/scenario/template.rb:6:in `perform'
from /builds/gitlab-org/gitlab-qa-mirror/lib/gitlab/qa/scenario/test/integration/geo.rb:73:in `block (4 levels) in perform'
from /builds/gitlab-org/gitlab-qa-mirror/lib/gitlab/qa/scenario/actable.rb:6:in `instance_exec'
from /builds/gitlab-org/gitlab-qa-mirror/lib/gitlab/qa/scenario/actable.rb:6:in `act'
from /builds/gitlab-org/gitlab-qa-mirror/lib/gitlab/qa/scenario/test/integration/geo.rb:63:in `block (3 levels) in perform'
from /builds/gitlab-org/gitlab-qa-mirror/lib/gitlab/qa/scenario/actable.rb:15:in `perform'
from /builds/gitlab-org/gitlab-qa-mirror/lib/gitlab/qa/scenario/test/integration/geo.rb:42:in `block (2 levels) in perform'
from /builds/gitlab-org/gitlab-qa-mirror/lib/gitlab/qa/component/base.rb:46:in `block in instance'
from /builds/gitlab-org/gitlab-qa-mirror/lib/gitlab/qa/component/base.rb:147:in `instance_no_teardown'
from /builds/gitlab-org/gitlab-qa-mirror/lib/gitlab/qa/component/base.rb:45:in `instance'
from /builds/gitlab-org/gitlab-qa-mirror/lib/gitlab/qa/scenario/test/integration/geo.rb:41:in `block in perform'
from /builds/gitlab-org/gitlab-qa-mirror/lib/gitlab/qa/scenario/actable.rb:15:in `perform'
from /builds/gitlab-org/gitlab-qa-mirror/lib/gitlab/qa/scenario/test/integration/geo.rb:19:in `perform'
from /builds/gitlab-org/gitlab-qa-mirror/lib/gitlab/qa/scenario/template.rb:8:in `block in perform'
from /builds/gitlab-org/gitlab-qa-mirror/lib/gitlab/qa/scenario/template.rb:6:in `tap'
from /builds/gitlab-org/gitlab-qa-mirror/lib/gitlab/qa/scenario/template.rb:6:in `perform'
from /builds/gitlab-org/gitlab-qa-mirror/lib/gitlab/qa/runner.rb:46:in `run'
from exe/gitlab-qa:8:in `<main>'Details of package version
Provide the package version installation details
Edited by Jennifer Louie