single-db jobs seem very flaky
In !108548 (closed) I experiment with disabling our automatic retry strategy as well as disabling our automatic flaky test skipping.
Interestingly, we have 23 failed jobs, all of them being single-db
jobs!
This issue is to investigate this fact.
Examples of failures:
1) Repositories::GitHttpController when repository container is a project behaves like Repositories::GitHttpController behaves like handles unavailable Gitaly when Gitaly is unavailable responds with a 503 message
Failure/Error: expect(response).to have_gitlab_http_status(:service_unavailable)
expected the response to have status code :service_unavailable but it was 200. The response was: {"GL_ID":"user-1711","GL_REPOSITORY":"project-1041","GL_USERNAME":"namespace976","ShowAllRefs":false,"Repository":{"storage_name":"default","relative_path":"@hashed/8c/6c/8c6c42f379f08f03b79653a3230abd5e8079999435030fd8ca703ae35fe9b37a.git","git_object_directory":"","git_alternate_object_directories":[],"gl_repository":"project-1041","gl_project_path":"namespace976/project979"},"GitConfigOptions":[],"GitalyServer":{"address":"unix:tmp/tests/gitaly/praefect.socket","token":"secret","call_metadata":{"user_id":"user-1711","username":"namespace976","remote_ip":"0.0.0.0"}}}
Click to expand
https://gitlab.com/gitlab-org/gitlab/-/jobs/3664694607 and exact same failure for the as-if-foss
job at https://gitlab.com/gitlab-org/gitlab/-/jobs/3664694887.
1) Repositories::GitHttpController when repository container is a project behaves like Repositories::GitHttpController behaves like handles unavailable Gitaly when Gitaly is unavailable responds with a 503 message
Failure/Error: expect(response).to have_gitlab_http_status(:service_unavailable)
expected the response to have status code :service_unavailable but it was 200. The response was: {"GL_ID":"user-1711","GL_REPOSITORY":"project-1041","GL_USERNAME":"namespace976","ShowAllRefs":false,"Repository":{"storage_name":"default","relative_path":"@hashed/8c/6c/8c6c42f379f08f03b79653a3230abd5e8079999435030fd8ca703ae35fe9b37a.git","git_object_directory":"","git_alternate_object_directories":[],"gl_repository":"project-1041","gl_project_path":"namespace976/project979"},"GitConfigOptions":[],"GitalyServer":{"address":"unix:tmp/tests/gitaly/praefect.socket","token":"secret","call_metadata":{"user_id":"user-1711","username":"namespace976","remote_ip":"0.0.0.0"}}}
Shared Example Group: "handles unavailable Gitaly" called from ./spec/controllers/repositories/git_http_controller_spec.rb:35
Shared Example Group: Repositories::GitHttpController called from ./spec/controllers/repositories/git_http_controller_spec.rb:30
# ./spec/controllers/repositories/git_http_controller_spec.rb:23:in `block (4 levels) in <top (required)>'
# ./spec/spec_helper.rb:512:in `block (3 levels) in <top (required)>'
# ./spec/support/sidekiq_middleware.rb:18:in `with_sidekiq_server_middleware'
# ./spec/spec_helper.rb:504:in `block (2 levels) in <top (required)>'
# ./spec/spec_helper.rb:500:in `block (3 levels) in <top (required)>'
# ./lib/gitlab/application_context.rb:59:in `with_raw_context'
# ./spec/spec_helper.rb:500:in `block (2 levels) in <top (required)>'
# ./spec/spec_helper.rb:239:in `block (2 levels) in <top (required)>'
# ./spec/support/system_exit_detected.rb:7:in `block (2 levels) in <top (required)>'
# ./spec/support/database/prevent_cross_joins.rb:106:in `block (3 levels) in <top (required)>'
# ./spec/support/database/prevent_cross_joins.rb:60:in `with_cross_joins_prevented'
# ./spec/support/database/prevent_cross_joins.rb:106:in `block (2 levels) in <top (required)>'
1.1) Failure/Error: expect(page).to have_content('Unresolve thread')
expected to find text "Unresolve thread" in "Sidney Jones86 @user56 · just now Maintainer Resolved by Sidney Jones86 just now Suggested change Applied 3 url = git://github.com/randx/six.git 4 [submodule \"gitlab-shell\"] 5 path = gitlab-shell 6 url = https://github.com/gitlabhq/gitlab-shell.git 7 [submodule \"gitlab-grack\"] 8 path = gitlab-grack 9 url = https://gitlab.com/gitlab-org/gitlab-grack.git 3 # change to a 4 # comment 5 # with 6 # broken 7 # lines Resolve thread"
Click to expand
https://gitlab.com/gitlab-org/gitlab/-/jobs/3664694686
1) User comments on a diff multi-line suggestions resolves discussion when applied
Got 1 failure and 1 other error:
1.1) Failure/Error: expect(page).to have_content('Unresolve thread')
expected to find text "Unresolve thread" in "Sidney Jones86 @user56 · just now Maintainer Resolved by Sidney Jones86 just now Suggested change Applied 3 url = git://github.com/randx/six.git 4 [submodule \"gitlab-shell\"] 5 path = gitlab-shell 6 url = https://github.com/gitlabhq/gitlab-shell.git 7 [submodule \"gitlab-grack\"] 8 path = gitlab-grack 9 url = https://gitlab.com/gitlab-org/gitlab-grack.git 3 # change to a 4 # comment 5 # with 6 # broken 7 # lines Resolve thread"
Timeout (30s) reached while running a waiting Capybara finder.
Consider using a non-waiting finder.
See https://www.cloudbees.com/blog/faster-rails-tests
# ./spec/features/merge_request/user_suggests_changes_on_diff_spec.rb:378:in `block (4 levels) in <top (required)>'
# ./spec/features/merge_request/user_suggests_changes_on_diff_spec.rb:371:in `block (3 levels) in <top (required)>'
# ./spec/spec_helper.rb:512:in `block (3 levels) in <top (required)>'
# ./spec/support/sidekiq_middleware.rb:18:in `with_sidekiq_server_middleware'
# ./spec/spec_helper.rb:504:in `block (2 levels) in <top (required)>'
# ./spec/spec_helper.rb:500:in `block (3 levels) in <top (required)>'
# ./lib/gitlab/application_context.rb:59:in `with_raw_context'
# ./spec/spec_helper.rb:500:in `block (2 levels) in <top (required)>'
# ./spec/spec_helper.rb:239:in `block (2 levels) in <top (required)>'
# ./spec/support/system_exit_detected.rb:7:in `block (2 levels) in <top (required)>'
# ./spec/support/database/prevent_cross_joins.rb:106:in `block (3 levels) in <top (required)>'
# ./spec/support/database/prevent_cross_joins.rb:60:in `with_cross_joins_prevented'
# ./spec/support/database/prevent_cross_joins.rb:106:in `block (2 levels) in <top (required)>'
1) Work item for signed in user behaves like work items assignees sucessfully assigns the current user by searching
Got 1 failure and 1 other error:
Shared Example Group: "work items assignees" called from ./spec/features/work_items/work_item_spec.rb:20
1.1) Failure/Error: expect(work_item.assignees).to include(user)
expected #<ActiveRecord::Associations::CollectionProxy [#<User id:307 @namespace75>]> to include #<User id:308 @user227>
Diff:
@@ -1 +1 @@
-[#<User id:308 @user227>]
+[#<User id:307 @namespace75>]
Click to expand
https://gitlab.com/gitlab-org/gitlab/-/jobs/3664694975
1) Work item for signed in user behaves like work items assignees sucessfully assigns the current user by searching
Got 1 failure and 1 other error:
Shared Example Group: "work items assignees" called from ./spec/features/work_items/work_item_spec.rb:20
1.1) Failure/Error: expect(work_item.assignees).to include(user)
expected #<ActiveRecord::Associations::CollectionProxy [#<User id:307 @namespace75>]> to include #<User id:308 @user227>
Diff:
@@ -1 +1 @@
-[#<User id:308 @user227>]
+[#<User id:307 @namespace75>]
# ./spec/support/shared_examples/features/work_items_shared_examples.rb:45:in `block (2 levels) in <top (required)>'
# ./spec/spec_helper.rb:512:in `block (3 levels) in <top (required)>'
# ./spec/support/sidekiq_middleware.rb:18:in `with_sidekiq_server_middleware'
# ./spec/spec_helper.rb:504:in `block (2 levels) in <top (required)>'
# ./spec/spec_helper.rb:500:in `block (3 levels) in <top (required)>'
# ./lib/gitlab/application_context.rb:59:in `with_raw_context'
# ./spec/spec_helper.rb:500:in `block (2 levels) in <top (required)>'
# ./spec/spec_helper.rb:239:in `block (2 levels) in <top (required)>'
# ./spec/support/system_exit_detected.rb:7:in `block (2 levels) in <top (required)>'
# ./spec/support/database/prevent_cross_joins.rb:106:in `block (3 levels) in <top (required)>'
# ./spec/support/database/prevent_cross_joins.rb:60:in `with_cross_joins_prevented'
# ./spec/support/database/prevent_cross_joins.rb:106:in `block (2 levels) in <top (required)>'
1) Group issues page manual ordering issues should not be draggable when user is not logged in
Got 1 failure and 1 other error:
1.1) Failure/Error: expect(page).to have_text 'An error occurred while reordering issues.'
expected to find text "An error occurred while reordering issues." in "Skip to content GitLab / Help Sign in / Register G group38 Group information Issues 3 List Board Milestones Merge requests 0 Packages and registries Collapse sidebar group38 Issues Open 3 Closed 0 All 3 Select project to create issue Toggle project select Toggle history Manual Issue #1 Issue group38/project126#1 · created just now by Sidney Jones205 updated just now Issue #2 Issue group38/project126#2 · created just now by Sidney Jones205 updated just now Issue #3 Issue group38/project126#3 · created just now by Sidney Jones205 updated just now"
Click to expand
https://gitlab.com/gitlab-org/gitlab/-/jobs/3664694994
1) Group issues page manual ordering issues should not be draggable when user is not logged in
Got 1 failure and 1 other error:
1.1) Failure/Error: expect(page).to have_text 'An error occurred while reordering issues.'
expected to find text "An error occurred while reordering issues." in "Skip to content GitLab / Help Sign in / Register G group38 Group information Issues 3 List Board Milestones Merge requests 0 Packages and registries Collapse sidebar group38 Issues Open 3 Closed 0 All 3 Select project to create issue Toggle project select Toggle history Manual Issue #1 Issue group38/project126#1 · created just now by Sidney Jones205 updated just now Issue #2 Issue group38/project126#2 · created just now by Sidney Jones205 updated just now Issue #3 Issue group38/project126#3 · created just now by Sidney Jones205 updated just now"
Timeout (30s) reached while running a waiting Capybara finder.
Consider using a non-waiting finder.
See https://www.cloudbees.com/blog/faster-rails-tests
# ./spec/features/groups/issues_spec.rb:189:in `block (3 levels) in <top (required)>'
# ./spec/spec_helper.rb:512:in `block (3 levels) in <top (required)>'
# ./spec/support/sidekiq_middleware.rb:18:in `with_sidekiq_server_middleware'
# ./spec/spec_helper.rb:504:in `block (2 levels) in <top (required)>'
# ./spec/spec_helper.rb:500:in `block (3 levels) in <top (required)>'
# ./lib/gitlab/application_context.rb:59:in `with_raw_context'
# ./spec/spec_helper.rb:500:in `block (2 levels) in <top (required)>'
# ./spec/spec_helper.rb:239:in `block (2 levels) in <top (required)>'
# ./spec/support/system_exit_detected.rb:7:in `block (2 levels) in <top (required)>'
# ./spec/support/database/prevent_cross_joins.rb:106:in `block (3 levels) in <top (required)>'
# ./spec/support/database/prevent_cross_joins.rb:60:in `with_cross_joins_prevented'
# ./spec/support/database/prevent_cross_joins.rb:106:in `block (2 levels) in <top (required)>'
1) Gitlab::Issuable::Clone::CopyResourceEventsService copies the resource label events
Failure/Error: raise CrossSchemaAccessError, message
Gitlab::Database::QueryAnalyzers::GitlabSchemasValidateConnection::CrossSchemaAccessError:
The query tried to access ["ci_namespace_mirrors"] (of gitlab_ci) which is outside of allowed schemas ([:gitlab_main, :gitlab_shared, :gitlab_internal, :gitlab_pm, :gitlab_main_clusterwide]) for the current connection 'main'
Click to expand
https://gitlab.com/gitlab-org/gitlab/-/jobs/3664694515
1) Gitlab::Issuable::Clone::CopyResourceEventsService copies the resource label events
Failure/Error: raise CrossSchemaAccessError, message
Gitlab::Database::QueryAnalyzers::GitlabSchemasValidateConnection::CrossSchemaAccessError:
The query tried to access ["ci_namespace_mirrors"] (of gitlab_ci) which is outside of allowed schemas ([:gitlab_main, :gitlab_shared, :gitlab_internal, :gitlab_pm, :gitlab_main_clusterwide]) for the current connection 'main'
# ./lib/gitlab/database/query_analyzers/gitlab_schemas_validate_connection.rb:34:in `analyze'
# ./lib/gitlab/database/query_analyzer.rb:89:in `block in process_sql'
# ./lib/gitlab/database/query_analyzer.rb:86:in `each'
# ./lib/gitlab/database/query_analyzer.rb:86:in `process_sql'
# ./lib/gitlab/database/query_analyzer.rb:28:in `block (2 levels) in hook!'
# ./lib/gitlab/database/query_analyzer.rb:114:in `with_ignored_recursive_calls'
# ./lib/gitlab/database/query_analyzer.rb:27:in `block in hook!'
# ./lib/gitlab/database/load_balancing/connection_proxy.rb:100:in `block in read_using_load_balancer'
# ./lib/gitlab/database/load_balancing/load_balancer.rb:127:in `block in read_write'
# ./lib/gitlab/database/load_balancing/load_balancer.rb:198:in `retry_with_backoff'
# ./lib/gitlab/database/load_balancing/load_balancer.rb:116:in `read_write'
# ./lib/gitlab/database/load_balancing/connection_proxy.rb:99:in `read_using_load_balancer'
# ./lib/gitlab/database/load_balancing/connection_proxy.rb:48:in `select_all'
# ./spec/factories/namespaces.rb:19:in `block (3 levels) in <top (required)>'
# <internal:kernel>:90:in `tap'
# <internal:kernel>:90:in `tap'
# <internal:kernel>:90:in `tap'
# <internal:kernel>:90:in `tap'
# ./spec/lib/gitlab/issuable/clone/copy_resource_events_service_spec.rb:16:in `block (2 levels) in <top (required)>'
# ./spec/spec_helper.rb:512:in `block (3 levels) in <top (required)>'
# ./spec/support/sidekiq_middleware.rb:18:in `with_sidekiq_server_middleware'
# ./spec/spec_helper.rb:504:in `block (2 levels) in <top (required)>'
# ./spec/spec_helper.rb:500:in `block (3 levels) in <top (required)>'
# ./lib/gitlab/application_context.rb:59:in `with_raw_context'
# ./spec/spec_helper.rb:500:in `block (2 levels) in <top (required)>'
# ./spec/spec_helper.rb:239:in `block (2 levels) in <top (required)>'
# ./spec/support/system_exit_detected.rb:7:in `block (2 levels) in <top (required)>'
# ./spec/support/database/prevent_cross_joins.rb:106:in `block (3 levels) in <top (required)>'
# ./spec/support/database/prevent_cross_joins.rb:60:in `with_cross_joins_prevented'
# ./spec/support/database/prevent_cross_joins.rb:106:in `block (2 levels) in <top (required)>'
1) Gitlab::Redis::RepositoryCache behaves like redis_shared_examples .version returns a version
Failure/Error: super(commands, &block)
Redis::CannotConnectError:
Error connecting to Redis on localhost:6380 (Errno::ECONNREFUSED)
Shared Example Group: "redis_shared_examples" called from ./spec/support/redis/redis_new_instance_shared_examples.rb:17
Shared Example Group: "redis_new_instance_shared_examples" called from ./spec/lib/gitlab/redis/repository_cache_spec.rb:6
Click to expand
https://gitlab.com/gitlab-org/gitlab/-/jobs/3664694506
1) Gitlab::Redis::RepositoryCache behaves like redis_shared_examples .version returns a version
Failure/Error: super(commands, &block)
Redis::CannotConnectError:
Error connecting to Redis on localhost:6380 (Errno::ECONNREFUSED)
Shared Example Group: "redis_shared_examples" called from ./spec/support/redis/redis_new_instance_shared_examples.rb:17
Shared Example Group: "redis_new_instance_shared_examples" called from ./spec/lib/gitlab/redis/repository_cache_spec.rb:6
# ./spec/spec_helper.rb:598:in `block in process'
# ./spec/spec_helper.rb:597:in `process'
# ./lib/gitlab/instrumentation/redis_interceptor.rb:10:in `block in call'
# ./lib/gitlab/instrumentation/redis_interceptor.rb:41:in `instrument_call'
# ./lib/gitlab/instrumentation/redis_interceptor.rb:9:in `call'
# ./lib/gitlab/redis/multi_store.rb:149:in `method_missing'
# ./lib/gitlab/redis/wrapper.rb:27:in `block in version'
# ./lib/gitlab/redis/wrapper.rb:23:in `block in with'
# ./lib/gitlab/redis/wrapper.rb:23:in `with'
# ./lib/gitlab/redis/wrapper.rb:27:in `version'
# ./spec/support/redis/redis_shared_examples.rb:242:in `block (3 levels) in <top (required)>'
# ./spec/spec_helper.rb:512:in `block (3 levels) in <top (required)>'
# ./spec/support/sidekiq_middleware.rb:18:in `with_sidekiq_server_middleware'
# ./spec/spec_helper.rb:504:in `block (2 levels) in <top (required)>'
# ./spec/spec_helper.rb:500:in `block (3 levels) in <top (required)>'
# ./lib/gitlab/application_context.rb:59:in `with_raw_context'
# ./spec/spec_helper.rb:500:in `block (2 levels) in <top (required)>'
# ./spec/spec_helper.rb:239:in `block (2 levels) in <top (required)>'
# ./spec/support/system_exit_detected.rb:7:in `block (2 levels) in <top (required)>'
# ./spec/support/database/prevent_cross_joins.rb:106:in `block (3 levels) in <top (required)>'
# ./spec/support/database/prevent_cross_joins.rb:60:in `with_cross_joins_prevented'
# ./spec/support/database/prevent_cross_joins.rb:106:in `block (2 levels) in <top (required)>'
# ------------------
# --- Caused by: ---
# IO::EINPROGRESSWaitWritable:
# Operation now in progress - connect(2) would block
# ./spec/spec_helper.rb:598:in `block in process'
This relates to #353647 (closed) and !82090 (diffs).
Edited by Rémy Coutable