Web hook edit page timeouts for group hook with 500 error
Request fails with a 500 error message when the user attempts to open the edit page for a webhook.
GitLab team members can find more details in the internal ticket: https://gitlab.zendesk.com/agent/tickets/348701
The logs show SQL timeout PG::QueryCanceled: ERROR: canceling statement due to statement timeout
Backtrace:
lib/gitlab/database/load_balancing/connection_proxy.rb:104:in `block in read_using_load_balancer'
lib/gitlab/database/load_balancing/load_balancer.rb:55:in `read'
lib/gitlab/database/load_balancing/connection_proxy.rb:103:in `read_using_load_balancer'
lib/gitlab/database/load_balancing/connection_proxy.rb:48:in `select_all'
app/views/shared/hook_logs/_recent_deliveries_table.html.haml:10
app/views/shared/hook_logs/_index.html.haml:11
ee/app/views/groups/hooks/edit.html.haml:20
app/controllers/application_controller.rb:142:in `render'
ee/lib/gitlab/ip_address_state.rb:10:in `with'
ee/app/controllers/ee/application_controller.rb:45:in `set_current_ip_address'
app/controllers/application_controller.rb:537:in `set_current_admin'
lib/gitlab/session.rb:11:in `with_session'
app/controllers/application_controller.rb:528:in `set_session_storage'
lib/gitlab/i18n.rb:107:in `with_locale'
lib/gitlab/i18n.rb:113:in `with_user_locale'
app/controllers/application_controller.rb:516:in `set_locale'
app/controllers/application_controller.rb:510:in `set_current_context'
ee/lib/omni_auth/strategies/group_saml.rb:41:in `other_phase'
lib/gitlab/metrics/elasticsearch_rack_middleware.rb:16:in `call'
lib/gitlab/middleware/memory_report.rb:13:in `call'
lib/gitlab/middleware/speedscope.rb:13:in `call'
lib/gitlab/database/load_balancing/rack_middleware.rb:23:in `call'
lib/gitlab/middleware/rails_queue_duration.rb:33:in `call'
lib/gitlab/metrics/rack_middleware.rb:16:in `block in call'
lib/gitlab/metrics/web_transaction.rb:46:in `run'
lib/gitlab/metrics/rack_middleware.rb:16:in `call'
lib/gitlab/jira/middleware.rb:19:in `call'
lib/gitlab/middleware/go.rb:20:in `call'
lib/gitlab/etag_caching/middleware.rb:21:in `call'
lib/gitlab/middleware/query_analyzer.rb:11:in `block in call'
lib/gitlab/database/query_analyzer.rb:37:in `within'
lib/gitlab/middleware/query_analyzer.rb:11:in `call'
lib/gitlab/middleware/multipart.rb:173:in `call'
lib/gitlab/middleware/read_only/controller.rb:50:in `call'
lib/gitlab/middleware/read_only.rb:18:in `call'
lib/gitlab/middleware/same_site_cookies.rb:27:in `call'
lib/gitlab/middleware/handle_malformed_strings.rb:21:in `call'
lib/gitlab/middleware/basic_health_check.rb:25:in `call'
lib/gitlab/middleware/handle_ip_spoof_attack_error.rb:25:in `call'
lib/gitlab/middleware/request_context.rb:21:in `call'
lib/gitlab/middleware/webhook_recursion_detection.rb:15:in `call'
config/initializers/fix_local_cache_middleware.rb:11:in `call'
lib/gitlab/middleware/compressed_json.rb:37:in `call'
lib/gitlab/middleware/rack_multipart_tempfile_factory.rb:19:in `call'
lib/gitlab/middleware/sidekiq_web_static.rb:20:in `call'
lib/gitlab/metrics/requests_rack_middleware.rb:77:in `call'
lib/gitlab/middleware/release_env.rb:13:in `call'
Logs in kibana: https://log.gprd.gitlab.net/goto/e9e6b6b0-6bfe-11ed-85ed-e7557b0a598c
The query is:
SELECT "web_hook_logs".* FROM "web_hook_logs" WHERE "web_hook_logs"."web_hook_id" = $1 AND "web_hook_logs"."created_at" BETWEEN $2 AND $3 ORDER BY "web_hook_logs"."created_at" DESC LIMIT $5 OFFSET $4
Proposal
- Create a new index.
The index should have suitable coverage for the query above - so something like the following:
create index on web_hook_logs (web_hook_id, created_at)
Edited by Alex Kalderimis