Advanced search throws error when AWS enabled and ES unreachable

Summary

Related zendesk ticket: https://gitlab.zendesk.com/agent/tickets/542461

Advanced search Admin UI throws a 500 error when the page is saved:

• The AWS checkbox is checked
• The Elasticsearch or OpenSearch backend is unreachable
• Elasticsearch indexed is checked

Steps to reproduce

1. Check Elasticsearch indexing
2. Check Use AWS OpenSearch Service with IAM credentials

What is the current bug behavior?

HTTP 500 error, UI is not loaded

What is the expected correct behavior?

The UI should load and report that the server is unreachable in an alert

Relevant logs and/or screenshots

Click to expand log

{"severity":"ERROR","time":"2024-06-21T19:47:13.824Z","correlation_id":"01J0Y5DD6YY18W8EB58F8HWHCJ","meta.caller_id":"Admin::ApplicationSettingsController#advanced_search","meta.remote_ip":"76.202.225.241","meta.feature_category":"global_search","meta.user":"root","meta.user_id":1,"meta.client_id":"user/1","exception.class":"Faraday::ConnectionFailed","exception.message":"Failed to open TCP connection to localhost:9200 (Connection refused - connect(2) for \"localhost\" port 9200)","exception.backtrace":["gems/gitlab-http/lib/net_http/connect_patch.rb:52:in `initialize'","gems/gitlab-http/lib/net_http/connect_patch.rb:52:in `open'","gems/gitlab-http/lib/net_http/connect_patch.rb:52:in `block in connect'","timeout (0.3.2) lib/timeout.rb:189:in `block in timeout'","timeout (0.3.2) lib/timeout.rb:196:in `timeout'","gems/gitlab-http/lib/net_http/connect_patch.rb:50:in `connect'","net-http (0.4.1) lib/net/http.rb:1580:in `do_start'","net-http (0.4.1) lib/net/http.rb:1569:in `start'","faraday-net_http (1.0.1) lib/faraday/adapter/net_http.rb:146:in `request_via_request_method'","faraday-net_http (1.0.1) lib/faraday/adapter/net_http.rb:131:in `request_with_wrapped_block'","faraday-net_http (1.0.1) lib/faraday/adapter/net_http.rb:122:in `perform_request'","faraday-net_http (1.0.1) lib/faraday/adapter/net_http.rb:66:in `block in call'","faraday (1.10.0) lib/faraday/adapter.rb:50:in `connection'","faraday-net_http (1.0.1) lib/faraday/adapter/net_http.rb:64:in `call'","faraday (1.10.0) lib/faraday/rack_builder.rb:154:in `build_response'","faraday (1.10.0) lib/faraday/connection.rb:516:in `run_request'","elasticsearch-transport (7.13.3) lib/elasticsearch/transport/transport/http/faraday.rb:48:in `block in perform_request'","elasticsearch-transport (7.13.3) lib/elasticsearch/transport/transport/base.rb:288:in `perform_request'","elasticsearch-transport (7.13.3) lib/elasticsearch/transport/transport/http/faraday.rb:37:in `perform_request'","elasticsearch-transport (7.13.3) lib/elasticsearch/transport/client.rb:192:in `perform_request'","lib/gitlab/instrumentation/elasticsearch_transport.rb:12:in `perform_request'","elasticsearch-api (7.13.3) lib/elasticsearch/api/namespace/common.rb:38:in `perform_request'","elasticsearch-api (7.13.3) lib/elasticsearch/api/actions/indices/exists_alias.rb:56:in `block in exists_alias'","elasticsearch-api (7.13.3) lib/elasticsearch/api/utils.rb:219:in `__rescue_from_not_found'","elasticsearch-api (7.13.3) lib/elasticsearch/api/actions/indices/exists_alias.rb:55:in `exists_alias'","ee/lib/gitlab/elastic/helper.rb:220:in `alias_exists?'","ee/lib/gitlab/elastic/helper.rb:224:in `alias_missing?'","ee/app/controllers/ee/admin/application_settings_controller.rb:48:in `elasticsearch_warn_if_not_using_aliases'","activesupport (7.0.8.4) lib/active_support/callbacks.rb:400:in `block in make_lambda'","activesupport (7.0.8.4) lib/active_support/callbacks.rb:180:in `block (2 levels) in halting_and_conditional'","actionpack (7.0.8.4) lib/abstract_controller/callbacks.rb:34:in `block (2 levels) in \u003cmodule:Callbacks\u003e'","activesupport (7.0.8.4) lib/active_support/callbacks.rb:181:in `block in halting_and_conditional'","activesupport (7.0.8.4) lib/active_support/callbacks.rb:595:in `block in invoke_before'","activesupport (7.0.8.4) lib/active_support/callbacks.rb:595:in `each'","activesupport (7.0.8.4) lib/active_support/callbacks.rb:595:in `invoke_before'","activesupport (7.0.8.4) lib/active_support/callbacks.rb:116:in `block in run_callbacks'","lib/gitlab/ip_address_state.rb:11:in `with'","ee/app/controllers/ee/application_controller.rb:45:in `set_current_ip_address'","activesupport (7.0.8.4) lib/active_support/callbacks.rb:127:in `block in run_callbacks'","app/controllers/application_controller.rb:468:in `set_current_admin'","activesupport (7.0.8.4) lib/active_support/callbacks.rb:127:in `block in run_callbacks'","lib/gitlab/session.rb:11:in `with_session'","app/controllers/application_controller.rb:459:in `set_session_storage'","activesupport (7.0.8.4) lib/active_support/callbacks.rb:127:in `block in run_callbacks'","lib/gitlab/i18n.rb:114:in `with_locale'","lib/gitlab/i18n.rb:120:in `with_user_locale'","app/controllers/application_controller.rb:450:in `set_locale'","activesupport (7.0.8.4) lib/active_support/callbacks.rb:127:in `block in run_callbacks'","marginalia (1.11.1) lib/marginalia.rb:109:in `record_query_comment'","activesupport (7.0.8.4) lib/active_support/callbacks.rb:127:in `block in run_callbacks'","app/controllers/application_controller.rb:443:in `set_current_context'","activesupport (7.0.8.4) lib/active_support/callbacks.rb:127:in `block in run_callbacks'","sentry-rails (5.17.3) lib/sentry/rails/controller_transaction.rb:28:in `block in sentry_around_action'","sentry-ruby (5.17.3) lib/sentry/hub.rb:102:in `with_child_span'","sentry-ruby (5.17.3) lib/sentry-ruby.rb:490:in `with_child_span'","sentry-rails (5.17.3) lib/sentry/rails/controller_transaction.rb:14:in `sentry_around_action'","activesupport (7.0.8.4) lib/active_support/callbacks.rb:127:in `block in run_callbacks'","activesupport (7.0.8.4) lib/active_support/callbacks.rb:138:in `run_callbacks'","actionpack (7.0.8.4) lib/abstract_controller/callbacks.rb:233:in `process_action'","actionpack (7.0.8.4) lib/action_controller/metal/rescue.rb:23:in `process_action'","actionpack (7.0.8.4) lib/action_controller/metal/instrumentation.rb:67:in `block in process_action'","activesupport (7.0.8.4) lib/active_support/notifications.rb:206:in `block in instrument'","activesupport (7.0.8.4) lib/active_support/notifications/instrumenter.rb:24:in `instrument'","activesupport (7.0.8.4) lib/active_support/notifications.rb:206:in `instrument'","actionpack (7.0.8.4) lib/action_controller/metal/instrumentation.rb:66:in `process_action'","actionpack (7.0.8.4) lib/action_controller/metal/params_wrapper.rb:259:in `process_action'","activerecord (7.0.8.4) lib/active_record/railties/controller_runtime.rb:27:in `process_action'","actionpack (7.0.8.4) lib/abstract_controller/base.rb:151:in `process'","actionview (7.0.8.4) lib/action_view/rendering.rb:39:in `process'","actionpack (7.0.8.4) lib/action_controller/metal.rb:188:in `dispatch'","actionpack (7.0.8.4) lib/action_controller/metal.rb:251:in `dispatch'","actionpack (7.0.8.4) lib/action_dispatch/routing/route_set.rb:49:in `dispatch'","actionpack (7.0.8.4) lib/action_dispatch/routing/route_set.rb:32:in `serve'","actionpack (7.0.8.4) lib/action_dispatch/journey/router.rb:50:in `block in serve'","actionpack (7.0.8.4) lib/action_dispatch/journey/router.rb:32:in `each'","actionpack (7.0.8.4) lib/action_dispatch/journey/router.rb:32:in `serve'","actionpack (7.0.8.4) lib/action_dispatch/routing/route_set.rb:852:in `call'","gitlab-experiment (0.9.1) lib/gitlab/experiment/middleware.rb:19:in `call'","flipper (0.26.2) lib/flipper/middleware/memoizer.rb:72:in `memoized_call'","flipper (0.26.2) lib/flipper/middleware/memoizer.rb:37:in `call'","lib/gitlab/metrics/elasticsearch_rack_middleware.rb:16:in `call'","lib/gitlab/middleware/sidekiq_shard_awareness_validation.rb:20:in `block in call'","lib/gitlab/sidekiq_sharding/validator.rb:42:in `enabled'","lib/gitlab/middleware/sidekiq_shard_awareness_validation.rb:20:in `call'","lib/gitlab/middleware/memory_report.rb:13:in `call'","lib/gitlab/middleware/speedscope.rb:13:in `call'","lib/gitlab/database/load_balancing/rack_middleware.rb:23:in `call'","lib/gitlab/middleware/rails_queue_duration.rb:33:in `call'","lib/gitlab/etag_caching/middleware.rb:21:in `call'","lib/gitlab/metrics/rack_middleware.rb:16:in `block in call'","lib/gitlab/metrics/web_transaction.rb:46:in `run'","lib/gitlab/metrics/rack_middleware.rb:16:in `call'","lib/gitlab/middleware/go.rb:20:in `call'","lib/gitlab/middleware/query_analyzer.rb:11:in `block in call'","lib/gitlab/database/query_analyzer.rb:40:in `within'","lib/gitlab/middleware/query_analyzer.rb:11:in `call'","lib/gitlab/middleware/organizations/current.rb:20:in `call'","batch-loader (2.0.5) lib/batch_loader/middleware.rb:11:in `call'","rack-attack (6.7.0) lib/rack/attack.rb:103:in `call'","apollo_upload_server (2.1.6) lib/apollo_upload_server/middleware.rb:19:in `call'","lib/gitlab/middleware/multipart.rb:173:in `call'","rack-attack (6.7.0) lib/rack/attack.rb:127:in `call'","warden (1.2.9) lib/warden/manager.rb:36:in `block in call'","warden (1.2.9) lib/warden/manager.rb:34:in `catch'","warden (1.2.9) lib/warden/manager.rb:34:in `call'","rack-cors (2.0.1) lib/rack/cors.rb:102:in `call'","rack (2.2.8.1) lib/rack/tempfile_reaper.rb:15:in `call'","rack (2.2.8.1) lib/rack/etag.rb:27:in `call'","rack (2.2.8.1) lib/rack/conditional_get.rb:40:in `call'","rack (2.2.8.1) lib/rack/head.rb:12:in `call'","actionpack (7.0.8.4) lib/action_dispatch/http/permissions_policy.rb:38:in `call'","actionpack (7.0.8.4) lib/action_dispatch/http/content_security_policy.rb:36:in `call'","lib/gitlab/middleware/read_only/controller.rb:50:in `call'","lib/gitlab/middleware/read_only.rb:18:in `call'","lib/gitlab/middleware/unauthenticated_session_expiry.rb:18:in `call'","rack (2.2.8.1) lib/rack/session/abstract/id.rb:266:in `context'","rack (2.2.8.1) lib/rack/session/abstract/id.rb:260:in `call'","actionpack (7.0.8.4) lib/action_dispatch/middleware/cookies.rb:704:in `call'","lib/gitlab/middleware/same_site_cookies.rb:27:in `call'","actionpack (7.0.8.4) lib/action_dispatch/middleware/callbacks.rb:27:in `block in call'","activesupport (7.0.8.4) lib/active_support/callbacks.rb:99:in `run_callbacks'","actionpack (7.0.8.4) lib/action_dispatch/middleware/callbacks.rb:26:in `call'","sentry-rails (5.17.3) lib/sentry/rails/rescued_exception_interceptor.rb:12:in `call'","actionpack (7.0.8.4) lib/action_dispatch/middleware/debug_exceptions.rb:28:in `call'","lib/gitlab/middleware/path_traversal_check.rb:27:in `call'","lib/gitlab/middleware/handle_malformed_strings.rb:21:in `call'","sentry-ruby (5.17.3) lib/sentry/rack/capture_exceptions.rb:29:in `block (2 levels) in call'","sentry-ruby (5.17.3) lib/sentry/hub.rb:251:in `with_session_tracking'","sentry-ruby (5.17.3) lib/sentry-ruby.rb:403:in `with_session_tracking'","sentry-ruby (5.17.3) lib/sentry/rack/capture_exceptions.rb:20:in `block in call'","sentry-ruby (5.17.3) lib/sentry/hub.rb:59:in `with_scope'","sentry-ruby (5.17.3) lib/sentry-ruby.rb:383:in `with_scope'","sentry-ruby (5.17.3) lib/sentry/rack/capture_exceptions.rb:19:in `call'","actionpack (7.0.8.4) lib/action_dispatch/middleware/show_exceptions.rb:29:in `call'","lib/gitlab/middleware/basic_health_check.rb:25:in `call'","lograge (0.11.2) lib/lograge/rails_ext/rack/logger.rb:15:in `call_app'","railties (7.0.8.4) lib/rails/rack/logger.rb:25:in `block in call'","activesupport (7.0.8.4) lib/active_support/tagged_logging.rb:99:in `block in tagged'","activesupport (7.0.8.4) lib/active_support/tagged_logging.rb:37:in `tagged'","activesupport (7.0.8.4) lib/active_support/tagged_logging.rb:99:in `tagged'","railties (7.0.8.4) lib/rails/rack/logger.rb:25:in `call'","actionpack (7.0.8.4) lib/action_dispatch/middleware/remote_ip.rb:93:in `call'","lib/gitlab/middleware/handle_ip_spoof_attack_error.rb:25:in `call'","lib/gitlab/middleware/request_context.rb:15:in `call'","lib/gitlab/middleware/webhook_recursion_detection.rb:15:in `call'","request_store (1.5.1) lib/request_store/middleware.rb:19:in `call'","rack (2.2.8.1) lib/rack/method_override.rb:24:in `call'","rack (2.2.8.1) lib/rack/runtime.rb:22:in `call'","rack-timeout (0.6.3) lib/rack/timeout/core.rb:148:in `block in call'","rack-timeout (0.6.3) lib/rack/timeout/support/timeout.rb:19:in `timeout'","rack-timeout (0.6.3) lib/rack/timeout/core.rb:147:in `call'","config/initializers/fix_local_cache_middleware.rb:11:in `call'","lib/gitlab/middleware/compressed_json.rb:44:in `call'","actionpack (7.0.8.4) lib/action_dispatch/middleware/executor.rb:14:in `call'","lib/gitlab/middleware/rack_multipart_tempfile_factory.rb:19:in `call'","rack (2.2.8.1) lib/rack/sendfile.rb:110:in `call'","lib/gitlab/middleware/sidekiq_web_static.rb:20:in `call'","lib/gitlab/metrics/requests_rack_middleware.rb:79:in `call'","gitlab-labkit (0.36.0) lib/labkit/middleware/rack.rb:22:in `block in call'","gitlab-labkit (0.36.0) lib/labkit/context.rb:35:in `with_context'","gitlab-labkit (0.36.0) lib/labkit/middleware/rack.rb:21:in `call'","actionpack (7.0.8.4) lib/action_dispatch/middleware/request_id.rb:26:in `call'","actionpack (7.0.8.4) lib/action_dispatch/middleware/host_authorization.rb:131:in `call'","railties (7.0.8.4) lib/rails/engine.rb:530:in `call'","railties (7.0.8.4) lib/rails/railtie.rb:226:in `public_send'","railties (7.0.8.4) lib/rails/railtie.rb:226:in `method_missing'","lib/gitlab/middleware/release_env.rb:13:in `call'","rack (2.2.8.1) lib/rack/urlmap.rb:74:in `block in call'","rack (2.2.8.1) lib/rack/urlmap.rb:58:in `each'","rack (2.2.8.1) lib/rack/urlmap.rb:58:in `call'","puma (6.4.0) lib/puma/configuration.rb:272:in `call'","puma (6.4.0) lib/puma/request.rb:100:in `block in handle_request'","puma (6.4.0) lib/puma/thread_pool.rb:378:in `with_force_shutdown'","puma (6.4.0) lib/puma/request.rb:99:in `handle_request'","puma (6.4.0) lib/puma/server.rb:443:in `process_client'","puma (6.4.0) lib/puma/server.rb:241:in `block in run'","puma (6.4.0) lib/puma/thread_pool.rb:155:in `block in spawn_thread'"],"exception.cause_class":"Errno::ECONNREFUSED","user.username":"root","tags.program":"web","tags.locale":"en","tags.feature_category":"global_search","tags.correlation_id":"01J0Y5DD6YY18W8EB58F8HWHCJ"}
{"severity":"ERROR","time":"2024-06-21T19:47:13.856Z","correlation_id":"01J0Y5DD6YY18W8EB58F8HWHCJ","meta.caller_id":"Admin::ApplicationSettingsController#advanced_search","meta.remote_ip":"76.202.225.241","meta.feature_category":"global_search","meta.user":"root","meta.user_id":1,"meta.client_id":"user/1","exception.class":"Faraday::ConnectionFailed","exception.message":"Failed to open TCP connection to localhost:9200 (Connection refused - connect(2) for \"localhost\" port 9200)","exception.backtrace":["gems/gitlab-http/lib/net_http/connect_patch.rb:52:in `initialize'","gems/gitlab-http/lib/net_http/connect_patch.rb:52:in `open'","gems/gitlab-http/lib/net_http/connect_patch.rb:52:in `block in connect'","timeout (0.3.2) lib/timeout.rb:189:in `block in timeout'","timeout (0.3.2) lib/timeout.rb:196:in `timeout'","gems/gitlab-http/lib/net_http/connect_patch.rb:50:in `connect'","net-http (0.4.1) lib/net/http.rb:1580:in `do_start'","net-http (0.4.1) lib/net/http.rb:1569:in `start'","faraday-net_http (1.0.1) lib/faraday/adapter/net_http.rb:146:in `request_via_request_method'","faraday-net_http (1.0.1) lib/faraday/adapter/net_http.rb:131:in `request_with_wrapped_block'","faraday-net_http (1.0.1) lib/faraday/adapter/net_http.rb:122:in `perform_request'","faraday-net_http (1.0.1) lib/faraday/adapter/net_http.rb:66:in `block in call'","faraday (1.10.0) lib/faraday/adapter.rb:50:in `connection'","faraday-net_http (1.0.1) lib/faraday/adapter/net_http.rb:64:in `call'","faraday (1.10.0) lib/faraday/rack_builder.rb:154:in `build_response'","faraday (1.10.0) lib/faraday/connection.rb:516:in `run_request'","elasticsearch-transport (7.13.3) lib/elasticsearch/transport/transport/http/faraday.rb:48:in `block in perform_request'","elasticsearch-transport (7.13.3) lib/elasticsearch/transport/transport/base.rb:288:in `perform_request'","elasticsearch-transport (7.13.3) lib/elasticsearch/transport/transport/http/faraday.rb:37:in `perform_request'","elasticsearch-transport (7.13.3) lib/elasticsearch/transport/client.rb:192:in `perform_request'","lib/gitlab/instrumentation/elasticsearch_transport.rb:12:in `perform_request'","elasticsearch-api (7.13.3) lib/elasticsearch/api/namespace/common.rb:38:in `perform_request'","elasticsearch-api (7.13.3) lib/elasticsearch/api/actions/indices/exists_alias.rb:56:in `block in exists_alias'","elasticsearch-api (7.13.3) lib/elasticsearch/api/utils.rb:219:in `__rescue_from_not_found'","elasticsearch-api (7.13.3) lib/elasticsearch/api/actions/indices/exists_alias.rb:55:in `exists_alias'","ee/lib/gitlab/elastic/helper.rb:220:in `alias_exists?'","ee/lib/gitlab/elastic/helper.rb:337:in `target_index_names'","ee/lib/gitlab/elastic/helper.rb:329:in `target_index_name'","ee/lib/gitlab/elastic/helper.rb:305:in `get_meta'","ee/app/controllers/ee/admin/application_settings_controller.rb:70:in `search_outdated_code_analyzer_detected'","activesupport (7.0.8.4) lib/active_support/callbacks.rb:400:in `block in make_lambda'","activesupport (7.0.8.4) lib/active_support/callbacks.rb:180:in `block (2 levels) in halting_and_conditional'","actionpack (7.0.8.4) lib/abstract_controller/callbacks.rb:34:in `block (2 levels) in \u003cmodule:Callbacks\u003e'","activesupport (7.0.8.4) lib/active_support/callbacks.rb:181:in `block in halting_and_conditional'","activesupport (7.0.8.4) lib/active_support/callbacks.rb:595:in `block in invoke_before'","activesupport (7.0.8.4) lib/active_support/callbacks.rb:595:in `each'","activesupport (7.0.8.4) lib/active_support/callbacks.rb:595:in `invoke_before'","activesupport (7.0.8.4) lib/active_support/callbacks.rb:116:in `block in run_callbacks'","lib/gitlab/ip_address_state.rb:11:in `with'","ee/app/controllers/ee/application_controller.rb:45:in `set_current_ip_address'","activesupport (7.0.8.4) lib/active_support/callbacks.rb:127:in `block in run_callbacks'","app/controllers/application_controller.rb:468:in `set_current_admin'","activesupport (7.0.8.4) lib/active_support/callbacks.rb:127:in `block in run_callbacks'","lib/gitlab/session.rb:11:in `with_session'","app/controllers/application_controller.rb:459:in `set_session_storage'","activesupport (7.0.8.4) lib/active_support/callbacks.rb:127:in `block in run_callbacks'","lib/gitlab/i18n.rb:114:in `with_locale'","lib/gitlab/i18n.rb:120:in `with_user_locale'","app/controllers/application_controller.rb:450:in `set_locale'","activesupport (7.0.8.4) lib/active_support/callbacks.rb:127:in `block in run_callbacks'","marginalia (1.11.1) lib/marginalia.rb:109:in `record_query_comment'","activesupport (7.0.8.4) lib/active_support/callbacks.rb:127:in `block in run_callbacks'","app/controllers/application_controller.rb:443:in `set_current_context'","activesupport (7.0.8.4) lib/active_support/callbacks.rb:127:in `block in run_callbacks'","sentry-rails (5.17.3) lib/sentry/rails/controller_transaction.rb:28:in `block in sentry_around_action'","sentry-ruby (5.17.3) lib/sentry/hub.rb:102:in `with_child_span'","sentry-ruby (5.17.3) lib/sentry-ruby.rb:490:in `with_child_span'","sentry-rails (5.17.3) lib/sentry/rails/controller_transaction.rb:14:in `sentry_around_action'","activesupport (7.0.8.4) lib/active_support/callbacks.rb:127:in `block in run_callbacks'","activesupport (7.0.8.4) lib/active_support/callbacks.rb:138:in `run_callbacks'","actionpack (7.0.8.4) lib/abstract_controller/callbacks.rb:233:in `process_action'","actionpack (7.0.8.4) lib/action_controller/metal/rescue.rb:23:in `process_action'","actionpack (7.0.8.4) lib/action_controller/metal/instrumentation.rb:67:in `block in process_action'","activesupport (7.0.8.4) lib/active_support/notifications.rb:206:in `block in instrument'","activesupport (7.0.8.4) lib/active_support/notifications/instrumenter.rb:24:in `instrument'","activesupport (7.0.8.4) lib/active_support/notifications.rb:206:in `instrument'","actionpack (7.0.8.4) lib/action_controller/metal/instrumentation.rb:66:in `process_action'","actionpack (7.0.8.4) lib/action_controller/metal/params_wrapper.rb:259:in `process_action'","activerecord (7.0.8.4) lib/active_record/railties/controller_runtime.rb:27:in `process_action'","actionpack (7.0.8.4) lib/abstract_controller/base.rb:151:in `process'","actionview (7.0.8.4) lib/action_view/rendering.rb:39:in `process'","actionpack (7.0.8.4) lib/action_controller/metal.rb:188:in `dispatch'","actionpack (7.0.8.4) lib/action_controller/metal.rb:251:in `dispatch'","actionpack (7.0.8.4) lib/action_dispatch/routing/route_set.rb:49:in `dispatch'","actionpack (7.0.8.4) lib/action_dispatch/routing/route_set.rb:32:in `serve'","actionpack (7.0.8.4) lib/action_dispatch/journey/router.rb:50:in `block in serve'","actionpack (7.0.8.4) lib/action_dispatch/journey/router.rb:32:in `each'","actionpack (7.0.8.4) lib/action_dispatch/journey/router.rb:32:in `serve'","actionpack (7.0.8.4) lib/action_dispatch/routing/route_set.rb:852:in `call'","gitlab-experiment (0.9.1) lib/gitlab/experiment/middleware.rb:19:in `call'","flipper (0.26.2) lib/flipper/middleware/memoizer.rb:72:in `memoized_call'","flipper (0.26.2) lib/flipper/middleware/memoizer.rb:37:in `call'","lib/gitlab/metrics/elasticsearch_rack_middleware.rb:16:in `call'","lib/gitlab/middleware/sidekiq_shard_awareness_validation.rb:20:in `block in call'","lib/gitlab/sidekiq_sharding/validator.rb:42:in `enabled'","lib/gitlab/middleware/sidekiq_shard_awareness_validation.rb:20:in `call'","lib/gitlab/middleware/memory_report.rb:13:in `call'","lib/gitlab/middleware/speedscope.rb:13:in `call'","lib/gitlab/database/load_balancing/rack_middleware.rb:23:in `call'","lib/gitlab/middleware/rails_queue_duration.rb:33:in `call'","lib/gitlab/etag_caching/middleware.rb:21:in `call'","lib/gitlab/metrics/rack_middleware.rb:16:in `block in call'","lib/gitlab/metrics/web_transaction.rb:46:in `run'","lib/gitlab/metrics/rack_middleware.rb:16:in `call'","lib/gitlab/middleware/go.rb:20:in `call'","lib/gitlab/middleware/query_analyzer.rb:11:in `block in call'","lib/gitlab/database/query_analyzer.rb:40:in `within'","lib/gitlab/middleware/query_analyzer.rb:11:in `call'","lib/gitlab/middleware/organizations/current.rb:20:in `call'","batch-loader (2.0.5) lib/batch_loader/middleware.rb:11:in `call'","rack-attack (6.7.0) lib/rack/attack.rb:103:in `call'","apollo_upload_server (2.1.6) lib/apollo_upload_server/middleware.rb:19:in `call'","lib/gitlab/middleware/multipart.rb:173:in `call'","rack-attack (6.7.0) lib/rack/attack.rb:127:in `call'","warden (1.2.9) lib/warden/manager.rb:36:in `block in call'","warden (1.2.9) lib/warden/manager.rb:34:in `catch'","warden (1.2.9) lib/warden/manager.rb:34:in `call'","rack-cors (2.0.1) lib/rack/cors.rb:102:in `call'","rack (2.2.8.1) lib/rack/tempfile_reaper.rb:15:in `call'","rack (2.2.8.1) lib/rack/etag.rb:27:in `call'","rack (2.2.8.1) lib/rack/conditional_get.rb:40:in `call'","rack (2.2.8.1) lib/rack/head.rb:12:in `call'","actionpack (7.0.8.4) lib/action_dispatch/http/permissions_policy.rb:38:in `call'","actionpack (7.0.8.4) lib/action_dispatch/http/content_security_policy.rb:36:in `call'","lib/gitlab/middleware/read_only/controller.rb:50:in `call'","lib/gitlab/middleware/read_only.rb:18:in `call'","lib/gitlab/middleware/unauthenticated_session_expiry.rb:18:in `call'","rack (2.2.8.1) lib/rack/session/abstract/id.rb:266:in `context'","rack (2.2.8.1) lib/rack/session/abstract/id.rb:260:in `call'","actionpack (7.0.8.4) lib/action_dispatch/middleware/cookies.rb:704:in `call'","lib/gitlab/middleware/same_site_cookies.rb:27:in `call'","actionpack (7.0.8.4) lib/action_dispatch/middleware/callbacks.rb:27:in `block in call'","activesupport (7.0.8.4) lib/active_support/callbacks.rb:99:in `run_callbacks'","actionpack (7.0.8.4) lib/action_dispatch/middleware/callbacks.rb:26:in `call'","sentry-rails (5.17.3) lib/sentry/rails/rescued_exception_interceptor.rb:12:in `call'","actionpack (7.0.8.4) lib/action_dispatch/middleware/debug_exceptions.rb:28:in `call'","lib/gitlab/middleware/path_traversal_check.rb:27:in `call'","lib/gitlab/middleware/handle_malformed_strings.rb:21:in `call'","sentry-ruby (5.17.3) lib/sentry/rack/capture_exceptions.rb:29:in `block (2 levels) in call'","sentry-ruby (5.17.3) lib/sentry/hub.rb:251:in `with_session_tracking'","sentry-ruby (5.17.3) lib/sentry-ruby.rb:403:in `with_session_tracking'","sentry-ruby (5.17.3) lib/sentry/rack/capture_exceptions.rb:20:in `block in call'","sentry-ruby (5.17.3) lib/sentry/hub.rb:59:in `with_scope'","sentry-ruby (5.17.3) lib/sentry-ruby.rb:383:in `with_scope'","sentry-ruby (5.17.3) lib/sentry/rack/capture_exceptions.rb:19:in `call'","actionpack (7.0.8.4) lib/action_dispatch/middleware/show_exceptions.rb:29:in `call'","lib/gitlab/middleware/basic_health_check.rb:25:in `call'","lograge (0.11.2) lib/lograge/rails_ext/rack/logger.rb:15:in `call_app'","railties (7.0.8.4) lib/rails/rack/logger.rb:25:in `block in call'","activesupport (7.0.8.4) lib/active_support/tagged_logging.rb:99:in `block in tagged'","activesupport (7.0.8.4) lib/active_support/tagged_logging.rb:37:in `tagged'","activesupport (7.0.8.4) lib/active_support/tagged_logging.rb:99:in `tagged'","railties (7.0.8.4) lib/rails/rack/logger.rb:25:in `call'","actionpack (7.0.8.4) lib/action_dispatch/middleware/remote_ip.rb:93:in `call'","lib/gitlab/middleware/handle_ip_spoof_attack_error.rb:25:in `call'","lib/gitlab/middleware/request_context.rb:15:in `call'","lib/gitlab/middleware/webhook_recursion_detection.rb:15:in `call'","request_store (1.5.1) lib/request_store/middleware.rb:19:in `call'","rack (2.2.8.1) lib/rack/method_override.rb:24:in `call'","rack (2.2.8.1) lib/rack/runtime.rb:22:in `call'","rack-timeout (0.6.3) lib/rack/timeout/core.rb:148:in `block in call'","rack-timeout (0.6.3) lib/rack/timeout/support/timeout.rb:19:in `timeout'","rack-timeout (0.6.3) lib/rack/timeout/core.rb:147:in `call'","config/initializers/fix_local_cache_middleware.rb:11:in `call'","lib/gitlab/middleware/compressed_json.rb:44:in `call'","actionpack (7.0.8.4) lib/action_dispatch/middleware/executor.rb:14:in `call'","lib/gitlab/middleware/rack_multipart_tempfile_factory.rb:19:in `call'","rack (2.2.8.1) lib/rack/sendfile.rb:110:in `call'","lib/gitlab/middleware/sidekiq_web_static.rb:20:in `call'","lib/gitlab/metrics/requests_rack_middleware.rb:79:in `call'","gitlab-labkit (0.36.0) lib/labkit/middleware/rack.rb:22:in `block in call'","gitlab-labkit (0.36.0) lib/labkit/context.rb:35:in `with_context'","gitlab-labkit (0.36.0) lib/labkit/middleware/rack.rb:21:in `call'","actionpack (7.0.8.4) lib/action_dispatch/middleware/request_id.rb:26:in `call'","actionpack (7.0.8.4) lib/action_dispatch/middleware/host_authorization.rb:131:in `call'","railties (7.0.8.4) lib/rails/engine.rb:530:in `call'","railties (7.0.8.4) lib/rails/railtie.rb:226:in `public_send'","railties (7.0.8.4) lib/rails/railtie.rb:226:in `method_missing'","lib/gitlab/middleware/release_env.rb:13:in `call'","rack (2.2.8.1) lib/rack/urlmap.rb:74:in `block in call'","rack (2.2.8.1) lib/rack/urlmap.rb:58:in `each'","rack (2.2.8.1) lib/rack/urlmap.rb:58:in `call'","puma (6.4.0) lib/puma/configuration.rb:272:in `call'","puma (6.4.0) lib/puma/request.rb:100:in `block in handle_request'","puma (6.4.0) lib/puma/thread_pool.rb:378:in `with_force_shutdown'","puma (6.4.0) lib/puma/request.rb:99:in `handle_request'","puma (6.4.0) lib/puma/server.rb:443:in `process_client'","puma (6.4.0) lib/puma/server.rb:241:in `block in run'","puma (6.4.0) lib/puma/thread_pool.rb:155:in `block in spawn_thread'"],"exception.cause_class":"Errno::ECONNREFUSED","user.username":"root","tags.program":"web","tags.locale":"en","tags.feature_category":"global_search","tags.correlation_id":"01J0Y5DD6YY18W8EB58F8HWHCJ"}

Possible fixes