ee:Geo test failing then timing out as taking longer than 2 hours
Summary
ee:Geo is failing then timing out as taking longer than 2 hours
See https://gitlab.com/gitlab-org/gitlab-qa-mirror/-/jobs/812457801
Stack trace
D, [2020-10-26T16:45:12.100450 #20] DEBUG -- : has_element? :user_avatar (wait: 0) returned: false
user renames project (FAILED - 2)
1st Try error in ./qa/specs/features/ee/browser_ui/geo/rename_replication_spec.rb:6:
Net::ReadTimeout
Net::ReadTimeout
Net::ReadTimeout
...
An error occurred in an `after(:context)` hook.
Failure/Error: if !QA::Runtime::Browser.blank_page? && QA::Page::Main::Menu.perform(&:signed_in?)
Net::ReadTimeout:
Net::ReadTimeout with #<TCPSocket:(closed)>
# /usr/local/bundle/gems/selenium-webdriver-3.142.6/lib/selenium/webdriver/remote/http/default.rb:129:in `response_for'
# /usr/local/bundle/gems/selenium-webdriver-3.142.6/lib/selenium/webdriver/remote/http/default.rb:82:in `request'
# /usr/local/bundle/gems/selenium-webdriver-3.142.6/lib/selenium/webdriver/remote/http/common.rb:64:in `call'
# /usr/local/bundle/gems/selenium-webdriver-3.142.6/lib/selenium/webdriver/remote/bridge.rb:167:in `execute'
# /usr/local/bundle/gems/selenium-webdriver-3.142.6/lib/selenium/webdriver/remote/w3c/bridge.rb:567:in `execute'
# /usr/local/bundle/gems/selenium-webdriver-3.142.6/lib/selenium/webdriver/remote/w3c/bridge.rb:558:in `find_elements_by'
# /usr/local/bundle/gems/selenium-webdriver-3.142.6/lib/selenium/webdriver/common/search_context.rb:80:in `find_elements'
# /usr/local/bundle/gems/capybara-3.29.0/lib/capybara/selenium/extensions/find.rb:17:in `find_by'
# /usr/local/bundle/gems/capybara-3.29.0/lib/capybara/selenium/extensions/find.rb:11:in `find_css'
# /usr/local/bundle/gems/capybara-3.29.0/lib/capybara/node/base.rb:107:in `find_css'
# /usr/local/bundle/gems/capybara-3.29.0/lib/capybara/queries/selector_query.rb:234:in `find_nodes_by_selector_format'
# /usr/local/bundle/gems/capybara-3.29.0/lib/capybara/queries/selector_query.rb:150:in `block in resolve_for'
# /usr/local/bundle/gems/capybara-3.29.0/lib/capybara/node/base.rb:77:in `synchronize'
# /usr/local/bundle/gems/capybara-3.29.0/lib/capybara/queries/selector_query.rb:149:in `resolve_for'
# /usr/local/bundle/gems/capybara-3.29.0/lib/capybara/node/matchers.rb:832:in `block in _verify_selector_result'
# /usr/local/bundle/gems/capybara-3.29.0/lib/capybara/node/base.rb:83:in `synchronize'
# /usr/local/bundle/gems/capybara-3.29.0/lib/capybara/node/matchers.rb:831:in `_verify_selector_result'
# /usr/local/bundle/gems/capybara-3.29.0/lib/capybara/node/matchers.rb:232:in `assert_no_selector'
# /usr/local/bundle/gems/capybara-3.29.0/lib/capybara/node/matchers.rb:51:in `block in has_no_selector?'
# /usr/local/bundle/gems/capybara-3.29.0/lib/capybara/node/matchers.rb:862:in `make_predicate'
# /usr/local/bundle/gems/capybara-3.29.0/lib/capybara/node/matchers.rb:51:in `has_no_selector?'
# /usr/local/bundle/gems/capybara-3.29.0/lib/capybara/node/matchers.rb:320:in `has_no_css?'
# /usr/local/bundle/gems/capybara-3.29.0/lib/capybara/session.rb:749:in `block (2 levels) in <class:Session>'
# ./qa/support/wait_for_requests.rb:28:in `block in finished_loading?'
# ./qa/support/retrier.rb:26:in `block in retry_on_exception'
# ./qa/support/repeater.rb:22:in `repeat_until'
# ./qa/support/retrier.rb:20:in `retry_on_exception'
# ./qa/support/wait_for_requests.rb:27:in `finished_loading?'
# ./qa/support/page/logging.rb:133:in `finished_loading?'
# ./qa/support/wait_for_requests.rb:12:in `block in wait_for_requests'
# ./qa/support/waiter.rb:31:in `block in wait_until'
# ./qa/support/repeater.rb:22:in `repeat_until'
# ./qa/support/waiter.rb:23:in `wait_until'
# ./qa/support/wait_for_requests.rb:11:in `wait_for_requests'
# ./qa/page/base.rb:181:in `has_element?'
# ./qa/support/page/logging.rb:98:in `has_element?'
# ./qa/page/main/menu.rb:129:in `has_personal_area?'
# ./qa/page/main/menu.rb:93:in `signed_in?'
# ./qa/scenario/actable.rb:16:in `perform'
# ./spec/spec_helper.rb:28:in `block (2 levels) in <top (required)>'
# ./qa/specs/runner.rb:71:in `perform'
# ./qa/scenario/template.rb:10:in `block in perform'
# ./qa/scenario/template.rb:8:in `tap'
# ./qa/scenario/template.rb:8:in `perform'
# ./qa/ee/scenario/test/geo.rb:44:in `perform'
# ./qa/scenario/template.rb:10:in `block in perform'
# ./qa/scenario/template.rb:8:in `tap'
# ./qa/scenario/template.rb:8:in `perform'
# ./qa/scenario/bootable.rb:28:in `launch!'
...
is replicated to the secondary (FAILED - 3)
1st Try error in ./qa/specs/features/ee/browser_ui/geo/ssh_push_spec.rb:15:
Net::ReadTimeout
Net::ReadTimeout
Net::ReadTimeout
...
remote:
Locking support detected on remote "ssh://git@gitlab-primary.geo/gitlab-qa-sandbox-group/qa-test-2020-10-26-16-26-42-443b077299a0c156/geo-project-45f1318237199b0c.git". Consider enabling it with:
$ git config lfs.https://gitlab-primary.geo/gitlab-qa-sandbox-group/qa-test-2020-10-26-16-26-42-443b077299a0c156/geo-project-45f1318237199b0c.git/info/lfs.locksverify true
remote: GitLab: 500 Internal Server Error
To ssh://gitlab-primary.geo/gitlab-qa-sandbox-group/qa-test-2020-10-26-16-26-42-443b077299a0c156/geo-project-45f1318237199b0c.git
! [remote rejected] master -> master (pre-receive hook declined)
error: failed to push some refs to 'ssh://git@gitlab-primary.geo/gitlab-qa-sandbox-group/qa-test-2020-10-26-16-26-42-443b077299a0c156/geo-project-45f1318237199b0c.git'], exitstatus=[1]
...
is replicated to the secondary (FAILED - 4)
1st Try error in ./qa/specs/features/ee/browser_ui/geo/ssh_push_spec.rb:77:
The command HOME="/tmp/qa-netrc-credentials/20" GIT_SSH_COMMAND="ssh -i /tmp/id_de78259bd4cdce8020201026-20-kn1g61 -o UserKnownHostsFile=/tmp/known_hosts_e263885c4034024920201026-20-1gztne6" git push ssh://git@gitlab-primary.geo/gitlab-qa-sandbox-group/qa-test-2020-10-26-16-26-42-443b077299a0c156/geo-project-45f1318237199b0c.git master:master 2>&1 failed (1) with the following output:
remote:
remote: INFO: Your SSH key is expiring soon. Please generate a new key.
remote:
Locking support detected on remote "ssh://git@gitlab-primary.geo/gitlab-qa-sandbox-group/qa-test-2020-10-26-16-26-42-443b077299a0c156/geo-project-45f1318237199b0c.git". Consider enabling it with:
$ git config lfs.https://gitlab-primary.geo/gitlab-qa-sandbox-group/qa-test-2020-10-26-16-26-42-443b077299a0c156/geo-project-45f1318237199b0c.git/info/lfs.locksverify true
remote: GitLab: 500 Internal Server Error
To ssh://gitlab-primary.geo/gitlab-qa-sandbox-group/qa-test-2020-10-26-16-26-42-443b077299a0c156/geo-project-45f1318237199b0c.git
! [remote rejected] master -> master (pre-receive hook declined)
error: failed to push some refs to 'ssh://git@gitlab-primary.geo/gitlab-qa-sandbox-group/qa-test-2020-10-26-16-26-42-443b077299a0c156/geo-project-45f1318237199b0c.git'
...
Starting test: Geo GitLab Geo HTTP push secondary git-lfs commit is redirected to the primary and ultimately replicated to the secondary
is redirected to the primary and ultimately replicated to the secondary (FAILED - 5)
1st Try error in ./qa/specs/features/ee/browser_ui/geo/http_push_to_secondary_spec.rb:92:
Net::ReadTimeout
Net::ReadTimeout
Net::ReadTimeout
...
Starting test: Geo GitLab Geo HTTP push secondary regular git commit is redirected to the primary and ultimately replicated to the secondary
is redirected to the primary and ultimately replicated to the secondary (FAILED - 6)
1st Try error in ./qa/specs/features/ee/browser_ui/geo/http_push_to_secondary_spec.rb:12:
Net::ReadTimeout
Net::ReadTimeout
Net::ReadTimeout
RSpec::Retry: 2nd try ./qa/specs/features/ee/browser_ui/geo/http_push_to_secondary_spec.rb:12
WARNING: Timed out waiting for logs to finish copying from container
...
ERROR: Job failed: execution took longer than 2h0m0s seconds