Multiple e2e tests failing in MRs targeting stable branches with "connections to all backends failing"
Summary
Multiple e2e tests are failing in MRs targeting 15-9-stable-ee
and 15-10-stable-ee
stable branches with "connections to all backends failing".
Example MR targeting 15-10-stable-ee
:
- https://gitlab.com/gitlab-org/security/gitlab/-/merge_requests/3220
- !118703 (merged)
- !118439 (merged)
Example MR targeting 15-9-stable-ee
:
Steps to reproduce
I was able to reproduce this using the Omnibus image created in one of the MRs:
sudo docker run \
--hostname localhost \
--publish 443:443 --publish 80:80 --publish 22:22 \
--name gitlab \
--platform linux/amd64 \
--env GITLAB_OMNIBUS_CONFIG="gitlab_rails['initial_root_password']='5iveL\!fe'; " \
--volume ~/srv/gitlab/config:/etc/gitlab \
--volume ~/srv/gitlab/logs:/var/log/gitlab \
--volume ~/srv/gitlab/data:/var/opt/gitlab \
registry.gitlab.com/gitlab-org/build/omnibus-gitlab-mirror/gitlab-ee:a255b57a43ef38408bdd07f7518f775882b04bbc
Login to the instance and a 500 is encountered.
Example Project
What is the current bug behavior?
Logging into a GitLab instance running 15-10-stable-ee
or 15-9-stable-ee
stable branch code fails
What is the expected correct behavior?
Logging into a GitLab instance running 15-10-stable-ee
or 15-9-stable-ee
stable branch code should work
Relevant logs and/or screenshots
ActionView::Template::Error (14:connections to all backends failing. debug_error_string:{"created":"@1682590196.097197839","description":"Failed to pick subchannel","file":"src/core/ext/filters/client_channel/client_channel.cc","file_line":3093,"referenced_errors":[{"created":"@1682590196.097191130","description":"connections to all backends failing","file":"src/core/lib/transport/error_utils.cc","file_line":163,"grpc_status":14}]}):
25: - if project.creator && use_creator_avatar
26: = render Pajamas::AvatarComponent.new(project.creator, size: 48, alt: '', class: 'gl-mr-5')
27: - else
28: = render Pajamas::AvatarComponent.new(project, size: 48, alt: '', class: 'gl-mr-5')
29: .project-cell{ class: css_class }
30: .project-details.gl-pr-9.gl-sm-pr-0.gl-w-full.gl-display-flex.gl-flex-direction-column{ data: { qa_selector: 'project_content', qa_project_name: project.name } }
31: .gl-display-flex.gl-align-items-center.gl-flex-wrap-wrap
lib/gitlab/gitaly_client.rb:185:in `execute'
lib/gitlab/gitaly_client/call.rb:18:in `block in call'
lib/gitlab/gitaly_client/call.rb:55:in `recording_request'
lib/gitlab/gitaly_client/call.rb:17:in `call'
lib/gitlab/gitaly_client.rb:175:in `call'
lib/gitlab/gitaly_client/with_feature_flag_actors.rb:31:in `block in gitaly_client_call'
lib/gitlab/gitaly_client.rb:541:in `with_feature_flag_actors'
lib/gitlab/gitaly_client/with_feature_flag_actors.rb:25:in `gitaly_client_call'
lib/gitlab/gitaly_client/repository_service.rb:22:in `exists?'
lib/gitlab/git/repository.rb:105:in `exists?'
app/models/repository.rb:563:in `exists?'
lib/gitlab/repository_cache_adapter.rb:95:in `block (2 levels) in cache_method_asymmetrically'
lib/gitlab/repository_cache.rb:44:in `fetch_without_caching_false'
lib/gitlab/repository_cache_adapter.rb:190:in `block (2 levels) in cache_method_output_asymmetrically'
lib/gitlab/safe_request_store.rb:12:in `fetch'
lib/gitlab/repository_cache.rb:25:in `fetch'
lib/gitlab/repository_cache_adapter.rb:189:in `block in cache_method_output_asymmetrically'
lib/gitlab/utils/strong_memoize.rb:34:in `strong_memoize'
lib/gitlab/repository_cache_adapter.rb:203:in `block in memoize_method_output'
lib/gitlab/repository_cache_adapter.rb:212:in `no_repository_fallback'
lib/gitlab/repository_cache_adapter.rb:202:in `memoize_method_output'
lib/gitlab/repository_cache_adapter.rb:188:in `cache_method_output_asymmetrically'
lib/gitlab/repository_cache_adapter.rb:94:in `block in cache_method_asymmetrically'
app/models/repository.rb:571:in `empty?'
app/models/repository.rb:696:in `tree'
app/models/repository.rb:1079:in `file_on_head'
app/models/repository.rb:607:in `block in avatar'
lib/gitlab/gitaly_client.rb:336:in `allow_n_plus_1_calls'
app/models/repository.rb:606:in `avatar'
lib/gitlab/repository_cache_adapter.rb:21:in `block (2 levels) in cache_method'
lib/gitlab/repository_cache.rb:25:in `fetch'
lib/gitlab/repository_cache_adapter.rb:163:in `block in cache_method_output'
lib/gitlab/utils/strong_memoize.rb:34:in `strong_memoize'
lib/gitlab/repository_cache_adapter.rb:203:in `block in memoize_method_output'
lib/gitlab/repository_cache_adapter.rb:212:in `no_repository_fallback'
lib/gitlab/repository_cache_adapter.rb:202:in `memoize_method_output'
lib/gitlab/repository_cache_adapter.rb:162:in `cache_method_output'
lib/gitlab/repository_cache_adapter.rb:20:in `block in cache_method'
app/models/project.rb:1696:in `avatar_in_git'
app/models/project.rb:1700:in `avatar_url'
app/models/concerns/avatarable.rb:36:in `avatar_url'
app/components/pajamas/avatar_component.rb:48:in `block in src'
lib/gitlab/utils/strong_memoize.rb:34:in `strong_memoize'
app/components/pajamas/avatar_component.rb:37:in `src'
app/components/pajamas/avatar_component.html.haml:1:in `call'
app/views/shared/projects/_project.html.haml:28
app/views/shared/projects/_project.html.haml:24
app/views/shared/projects/_project.html.haml:21
app/views/shared/projects/_list.html.haml:38
app/views/shared/projects/_list.html.haml:36:in `each_with_index'
app/views/shared/projects/_list.html.haml:36
app/views/dashboard/projects/_projects.html.haml:1
app/views/dashboard/projects/index.html.haml:16
app/controllers/application_controller.rb:142:in `render'
app/controllers/dashboard/projects_controller.rb:64:in `block in render_projects'
lib/gitlab/gitaly_client.rb:336:in `allow_n_plus_1_calls'
app/controllers/dashboard/projects_controller.rb:63:in `render_projects'
app/controllers/dashboard/projects_controller.rb:23:in `block (2 levels) in index'
app/controllers/dashboard/projects_controller.rb:21:in `index'
app/controllers/root_controller.rb:25:in `block in index'
lib/gitlab/gitaly_client.rb:336:in `allow_n_plus_1_calls'
app/controllers/root_controller.rb:23:in `index'
ee/lib/gitlab/ip_address_state.rb:10:in `with'
ee/app/controllers/ee/application_controller.rb:46:in `set_current_ip_address'
app/controllers/application_controller.rb:524:in `set_current_admin'
lib/gitlab/session.rb:11:in `with_session'
app/controllers/application_controller.rb:515:in `set_session_storage'
lib/gitlab/i18n.rb:107:in `with_locale'
lib/gitlab/i18n.rb:113:in `with_user_locale'
app/controllers/application_controller.rb:506:in `set_locale'
app/controllers/application_controller.rb:499:in `set_current_context'
lib/gitlab/metrics/elasticsearch_rack_middleware.rb:16:in `call'
lib/gitlab/middleware/memory_report.rb:13:in `call'
lib/gitlab/middleware/speedscope.rb:13:in `call'
lib/gitlab/database/load_balancing/rack_middleware.rb:23:in `call'
lib/gitlab/middleware/rails_queue_duration.rb:33:in `call'
lib/gitlab/metrics/rack_middleware.rb:16:in `block in call'
lib/gitlab/metrics/web_transaction.rb:46:in `run'
lib/gitlab/metrics/rack_middleware.rb:16:in `call'
lib/gitlab/jira/middleware.rb:19:in `call'
lib/gitlab/middleware/go.rb:20:in `call'
lib/gitlab/etag_caching/middleware.rb:21:in `call'
lib/gitlab/middleware/query_analyzer.rb:11:in `block in call'
lib/gitlab/database/query_analyzer.rb:37:in `within'
lib/gitlab/middleware/query_analyzer.rb:11:in `call'
lib/gitlab/middleware/multipart.rb:173:in `call'
lib/gitlab/middleware/read_only/controller.rb:50:in `call'
lib/gitlab/middleware/read_only.rb:18:in `call'
lib/gitlab/middleware/same_site_cookies.rb:27:in `call'
lib/gitlab/middleware/basic_health_check.rb:25:in `call'
lib/gitlab/middleware/handle_malformed_strings.rb:21:in `call'
lib/gitlab/middleware/handle_ip_spoof_attack_error.rb:25:in `call'
lib/gitlab/middleware/request_context.rb:21:in `call'
lib/gitlab/middleware/webhook_recursion_detection.rb:15:in `call'
config/initializers/fix_local_cache_middleware.rb:11:in `call'
lib/gitlab/middleware/compressed_json.rb:37:in `call'
lib/gitlab/middleware/rack_multipart_tempfile_factory.rb:19:in `call'
lib/gitlab/middleware/sidekiq_web_static.rb:20:in `call'
lib/gitlab/metrics/requests_rack_middleware.rb:79:in `call'
lib/gitlab/middleware/release_env.rb:13:in `call'
Possible fixes
Edited by Sanad Liaquat