PG::QueryCanceled: ERROR: "canceling statement due to statement timeout" on accessing group's settings > general
Summary
Visiting settings > general on any group results in a 500 - PG::QueryCanceled: ERROR: "canceling statement due to statement timeout"
eg: https://staging.gitlab.com/groups/test-group-top-level-500/-/edit
e2e test Job: https://ops.gitlab.net/gitlab-org/quality/staging/-/jobs/5849978
Stack trace
PG::QueryCanceled: ERROR: canceling statement due to statement timeout
ActionView::Template::Error
lib/gitlab/database/load_balancing/connection_proxy.rb:110:in `block in read_using_load_balancer', lib/gitlab/database/load_balancing/load_balancer.rb:55:in `read', lib/gitlab/database/load_balancing/connection_proxy.rb:109:in `read_using_load_balancer', lib/gitlab/database/load_balancing/connection_proxy.rb:54:in `select_all', app/helpers/groups_helper.rb:118:in `sort_by', app/helpers/groups_helper.rb:118:in `parent_group_options', app/views/groups/settings/_transfer.html.haml:15, app/views/groups/settings/_transfer.html.haml:4, app/views/groups/settings/_advanced.html.haml:28, app/views/groups/edit.html.haml:59, app/controllers/application_controller.rb:133:in `render', ee/lib/gitlab/ip_address_state.rb:10:in `with', ee/app/controllers/ee/application_controller.rb:44:in `set_current_ip_address', app/controllers/application_controller.rb:490:in `set_current_admin', lib/gitlab/session.rb:11:in `with_session', app/controllers/application_controller.rb:481:in `set_session_storage', lib/gitlab/i18n.rb:105:in `with_locale', lib/gitlab/i18n.rb:111:in `with_user_locale', app/controllers/application_controller.rb:475:in `set_locale', app/controllers/application_controller.rb:469:in `set_current_context', ee/lib/omni_auth/strategies/group_saml.rb:41:in `other_phase', lib/gitlab/metrics/elasticsearch_rack_middleware.rb:16:in `call', lib/gitlab/middleware/rails_queue_duration.rb:33:in `call', lib/gitlab/middleware/speedscope.rb:13:in `call', lib/gitlab/request_profiler/middleware.rb:17:in `call', lib/gitlab/database/load_balancing/rack_middleware.rb:23:in `call', lib/gitlab/metrics/rack_middleware.rb:16:in `block in call', lib/gitlab/metrics/web_transaction.rb:46:in `run', lib/gitlab/metrics/rack_middleware.rb:16:in `call', lib/gitlab/jira/middleware.rb:19:in `call', lib/gitlab/middleware/go.rb:20:in `call', lib/gitlab/etag_caching/middleware.rb:21:in `call', lib/gitlab/middleware/query_analyzer.rb:11:in `block in call', lib/gitlab/database/query_analyzer.rb:42:in `within', lib/gitlab/middleware/query_analyzer.rb:11:in `call', lib/gitlab/middleware/multipart.rb:173:in `call', lib/gitlab/middleware/read_only/controller.rb:50:in `call', lib/gitlab/middleware/read_only.rb:18:in `call', lib/gitlab/middleware/same_site_cookies.rb:27:in `call', lib/gitlab/middleware/handle_malformed_strings.rb:21:in `call', lib/gitlab/middleware/basic_health_check.rb:25:in `call', lib/gitlab/middleware/handle_ip_spoof_attack_error.rb:25:in `call', lib/gitlab/middleware/request_context.rb:21:in `call', config/initializers/fix_local_cache_middleware.rb:11:in `call', lib/gitlab/middleware/compressed_json.rb:26:in `call', lib/gitlab/middleware/rack_multipart_tempfile_factory.rb:19:in `call', lib/gitlab/middleware/sidekiq_web_static.rb:20:in `call', lib/gitlab/metrics/requests_rack_middleware.rb:75:in `call', lib/gitlab/middleware/release_env.rb:13:in `call'
E2E test Stack trace
Failures:
1) Manage Group file templates creates file via custom LICENSE file template
Failure/Error: general.choose_file_template_repository(@file_template_project.name)
Capybara::ElementNotFound:
Unable to find css "[data-qa-selector=\"file_template_repositories\"],.qa-file-template-repositories"
# /usr/local/bundle/gems/capybara-3.35.3/lib/capybara/node/finders.rb:303:in `block in synced_resolve'
# /usr/local/bundle/gems/capybara-3.35.3/lib/capybara/node/base.rb:83:in `synchronize'
# /usr/local/bundle/gems/capybara-3.35.3/lib/capybara/node/finders.rb:292:in `synced_resolve'
# /usr/local/bundle/gems/capybara-3.35.3/lib/capybara/node/finders.rb:53:in `find'
# /usr/local/bundle/gems/capybara-3.35.3/lib/capybara/session.rb:762:in `find'
# /usr/local/bundle/gems/capybara-3.35.3/lib/capybara/session.rb:350:in `within'
# ./qa/page/base.rb:343:in `within_element'
# ./qa/support/page/logging.rb:149:in `within_element'
# ./qa/page/settings/common.rb:11:in `expand_content'
# ./qa/ee/page/group/settings/general.rb:120:in `choose_file_template_repository'
# ./qa/specs/features/ee/browser_ui/1_manage/group/group_file_template_spec.rb:117:in `block in set_file_template_if_not_already_set'
# ./qa/scenario/actable.rb:16:in `perform'
# ./qa/specs/features/ee/browser_ui/1_manage/group/group_file_template_spec.rb:116:in `set_file_template_if_not_already_set'
# ./qa/specs/features/ee/browser_ui/1_manage/group/group_file_template_spec.rb:84:in `block (4 levels) in <module:QA>'
Screenshot / HTML page
Screenshot shows 500
Possible fixes
Availability and Testing
Suggested test plan:
- Populate the MR’s review app with at least the same number of groups that
gitlab-qa-bot
is a member of on staging (880 top-level groups as of 2/23/22)- A custom script can be created that generates groups via the
/groups
API using the admin’s PAT. The admin will then be added as owner of these groups by default, and we can manually log in as admin to visit Settings -> General for any of the groups. - Or, once the initial
review-qa-*
jobs have completed, the script can be extended to invite one of thegitlab-qa-user
to each group (gitlab-qa-user1
seems to be the most widely used). We can then try rerunning the review app tests (review-qa-all
) again to see if any issues occur.
- A custom script can be created that generates groups via the
Please note pipeline:run-review-app
label may need to be applied to the MR if it is not automatically triggered by any of the conditions listed here
Feel free to reach out to @vburton to coordinate or discuss other data that may be needed
Edited by Valerie Burton