Flaky spec: spec/workers/container_registry/migration/enqueuer_worker_spec.rb
The following spec failures surfaced in broken master incidents
- gitlab-org/quality/engineering-productivity/master-broken-incidents#294 (closed)
- gitlab-org/quality/engineering-productivity/master-broken-incidents#305 (closed)
Failed examples:
spec/workers/container_registry/migration/enqueuer_worker_spec.rb:101
spec/workers/container_registry/migration/enqueuer_worker_spec.rb:182
spec/workers/container_registry/migration/enqueuer_worker_spec.rb:214
Failures:
1) ContainerRegistry::Migration::EnqueuerWorker#perform when multiple aborted imports are available when an error occurs does abort that migration
Failure/Error: logger.info(structured_payload(extras))
#<Gitlab::SidekiqLogging::ClientLogger:0x00007f1b38b87c98 @level=0, @progname=nil, @default_formatter=#<Logger::Formatter:0x00007f1b38b87bf8 @datetime_format=nil>, @formatter=#<Gitlab::SidekiqLogging::JSONFormatter:0x00007f1b38b5bbc0>, @logdev=#<Logger::LogDevice:0x00007f1b38b87b80 @shift_period_suffix="%Y%m%d", @shift_size=1048576, @shift_age=0, @filename=#<Pathname:/builds/gitlab-org/gitlab/log/sidekiq_client.log>, @dev=#<File:/builds/gitlab-org/gitlab/log/sidekiq_client.log>, @binmode=false, @mon_data=#<Monitor:0x00007f1b38b87ae0>, @mon_data_owner_object_id=216720>> received :info with unexpected arguments
expected: ({"class"=>"ContainerRegistry::Migration::EnqueuerWorker", "container_repository_id"=>79, "container_r...try", "jid"=>nil, "job_status"=>"running", "queue"=>"cronjob:container_registry_migration_enqueuer"})
got: ({"class"=>"ContainerRegistry::Migration::EnqueuerWorker", "container_repository_id"=>80, "container_r...try", "jid"=>nil, "job_status"=>"running", "queue"=>"cronjob:container_registry_migration_enqueuer"})
Diff:
@@ -1,7 +1,7 @@
[{"class"=>"ContainerRegistry::Migration::EnqueuerWorker",
- "container_repository_id"=>79,
+ "container_repository_id"=>80,
"container_repository_migration_state"=>"import_aborted",
- "container_repository_path"=>"namespace4/project4/test_image_4",
+ "container_repository_path"=>"namespace5/project5/test_image_5",
"correlation_id"=>"c7c1f4e18f252a818fa75c7c633b02cb",
"import_type"=>"retry",
"jid"=>nil,
# ./app/workers/container_registry/migration/enqueuer_worker.rb:160:in `log_info'
# ./app/workers/container_registry/migration/enqueuer_worker.rb:176:in `log_repository_info'
# ./app/workers/container_registry/migration/enqueuer_worker.rb:52:in `handle_aborted_migration'
# ./app/workers/container_registry/migration/enqueuer_worker.rb:22:in `block in perform'
# ./app/services/concerns/exclusive_lease_guard.rb:29:in `try_obtain_lease'
# ./app/workers/container_registry/migration/enqueuer_worker.rb:20:in `perform'
# ./spec/workers/container_registry/migration/enqueuer_worker_spec.rb:22:in `block (3 levels) in <main>'
# ./spec/workers/container_registry/migration/enqueuer_worker_spec.rb:117:in `block (5 levels) in <main>'
# ./spec/spec_helper.rb:413:in `block (3 levels) in <top (required)>'
# ./spec/support/sidekiq_middleware.rb:18:in `with_sidekiq_server_middleware'
# ./spec/spec_helper.rb:405:in `block (2 levels) in <top (required)>'
# ./spec/spec_helper.rb:401:in `block (3 levels) in <top (required)>'
# ./lib/gitlab/application_context.rb:59:in `with_raw_context'
# ./spec/spec_helper.rb:401:in `block (2 levels) in <top (required)>'
# ./spec/spec_helper.rb:241:in `block (2 levels) in <top (required)>'
# ./spec/support/system_exit_detected.rb:7:in `block (2 levels) in <main>'
# ./spec/support/redis.rb:17:in `block (3 levels) in <main>'
# ./spec/support/flaky_tests.rb:27:in `block (2 levels) in <main>'
# ./spec/support/database/prevent_cross_joins.rb:106:in `block (3 levels) in <main>'
# ./spec/support/database/prevent_cross_joins.rb:60:in `with_cross_joins_prevented'
# ./spec/support/database/prevent_cross_joins.rb:106:in `block (2 levels) in <main>'
2) ContainerRegistry::Migration::EnqueuerWorker#perform when multiple qualified repositories are available when the new pre-import maxes out the capacity starts the pre-import only for one qualified repository
Got 3 failures:
2.1) Failure/Error: logger.info(structured_payload(extras))
#<Gitlab::SidekiqLogging::ClientLogger:0x00007f1b38b87c98 @level=0, @progname=nil, @default_formatter=#<Logger::Formatter:0x00007f1b38b87bf8 @datetime_format=nil>, @formatter=#<Gitlab::SidekiqLogging::JSONFormatter:0x00007f1b38b5bbc0>, @logdev=#<Logger::LogDevice:0x00007f1b38b87b80 @shift_period_suffix="%Y%m%d", @shift_size=1048576, @shift_age=0, @filename=#<Pathname:/builds/gitlab-org/gitlab/log/sidekiq_client.log>, @dev=#<File:/builds/gitlab-org/gitlab/log/sidekiq_client.log>, @binmode=false, @mon_data=#<Monitor:0x00007f1b38b87ae0>, @mon_data_owner_object_id=216720>> received :info with unexpected arguments
expected: ({"class"=>"ContainerRegistry::Migration::EnqueuerWorker", "container_repository_id"=>76, "container_r...ext", "jid"=>nil, "job_status"=>"running", "queue"=>"cronjob:container_registry_migration_enqueuer"})
got: ({"class"=>"ContainerRegistry::Migration::EnqueuerWorker", "container_repository_id"=>81, "container_r...ext", "jid"=>nil, "job_status"=>"running", "queue"=>"cronjob:container_registry_migration_enqueuer"})
Diff:
@@ -1,7 +1,7 @@
[{"class"=>"ContainerRegistry::Migration::EnqueuerWorker",
- "container_repository_id"=>76,
+ "container_repository_id"=>81,
"container_repository_migration_state"=>"pre_importing",
- "container_repository_path"=>"namespace1/project1/test_image_1",
+ "container_repository_path"=>"namespace6/project6/test_image_6",
"correlation_id"=>"35e539e865bec1ebe4f52b06e6420de6",
"import_type"=>"next",
"jid"=>nil,
# ./app/workers/container_registry/migration/enqueuer_worker.rb:160:in `log_info'
# ./app/workers/container_registry/migration/enqueuer_worker.rb:176:in `log_repository_info'
# ./app/workers/container_registry/migration/enqueuer_worker.rb:69:in `handle_next_migration'
# ./app/workers/container_registry/migration/enqueuer_worker.rb:22:in `block in perform'
# ./app/services/concerns/exclusive_lease_guard.rb:29:in `try_obtain_lease'
# ./app/workers/container_registry/migration/enqueuer_worker.rb:20:in `perform'
# ./spec/workers/container_registry/migration/enqueuer_worker_spec.rb:22:in `block (3 levels) in <main>'
# ./spec/workers/container_registry/migration/enqueuer_worker_spec.rb:194:in `block (5 levels) in <main>'
# ./spec/spec_helper.rb:413:in `block (3 levels) in <top (required)>'
# ./spec/support/sidekiq_middleware.rb:18:in `with_sidekiq_server_middleware'
# ./spec/spec_helper.rb:405:in `block (2 levels) in <top (required)>'
# ./spec/spec_helper.rb:401:in `block (3 levels) in <top (required)>'
# ./lib/gitlab/application_context.rb:59:in `with_raw_context'
# ./spec/spec_helper.rb:401:in `block (2 levels) in <top (required)>'
# ./spec/spec_helper.rb:241:in `block (2 levels) in <top (required)>'
# ./spec/support/system_exit_detected.rb:7:in `block (2 levels) in <main>'
# ./spec/support/redis.rb:17:in `block (3 levels) in <main>'
# ./spec/support/flaky_tests.rb:27:in `block (2 levels) in <main>'
# ./spec/support/database/prevent_cross_joins.rb:106:in `block (3 levels) in <main>'
# ./spec/support/database/prevent_cross_joins.rb:60:in `with_cross_joins_prevented'
# ./spec/support/database/prevent_cross_joins.rb:106:in `block (2 levels) in <main>'
2.2) Failure/Error: expect(container_repository.reload).to be_pre_importing
expected `#<ContainerRepository id: 76, project_id: 704, name: "test_image_1", created_at: "2022-11-29 02:11:30...ion_plan: nil, last_cleanup_deleted_tags_count: nil, delete_started_at: nil, status_updated_at: nil>.pre_importing?` to be truthy, got false
# ./spec/workers/container_registry/migration/enqueuer_worker_spec.rb:196:in `block (5 levels) in <main>'
# ./spec/spec_helper.rb:413:in `block (3 levels) in <top (required)>'
# ./spec/support/sidekiq_middleware.rb:18:in `with_sidekiq_server_middleware'
# ./spec/spec_helper.rb:405:in `block (2 levels) in <top (required)>'
# ./spec/spec_helper.rb:401:in `block (3 levels) in <top (required)>'
# ./lib/gitlab/application_context.rb:59:in `with_raw_context'
# ./spec/spec_helper.rb:401:in `block (2 levels) in <top (required)>'
# ./spec/spec_helper.rb:241:in `block (2 levels) in <top (required)>'
# ./spec/support/system_exit_detected.rb:7:in `block (2 levels) in <main>'
# ./spec/support/redis.rb:17:in `block (3 levels) in <main>'
# ./spec/support/flaky_tests.rb:27:in `block (2 levels) in <main>'
# ./spec/support/database/prevent_cross_joins.rb:106:in `block (3 levels) in <main>'
# ./spec/support/database/prevent_cross_joins.rb:60:in `with_cross_joins_prevented'
# ./spec/support/database/prevent_cross_joins.rb:106:in `block (2 levels) in <main>'
2.3) Failure/Error: expect(container_repository2.reload).to be_default
expected `#<ContainerRepository id: 81, project_id: 709, name: "test_image_6", created_at: "2022-11-29 02:11:32...ion_plan: nil, last_cleanup_deleted_tags_count: nil, delete_started_at: nil, status_updated_at: nil>.default?` to be truthy, got false
# ./spec/workers/container_registry/migration/enqueuer_worker_spec.rb:197:in `block (5 levels) in <main>'
# ./spec/spec_helper.rb:413:in `block (3 levels) in <top (required)>'
# ./spec/support/sidekiq_middleware.rb:18:in `with_sidekiq_server_middleware'
# ./spec/spec_helper.rb:405:in `block (2 levels) in <top (required)>'
# ./spec/spec_helper.rb:401:in `block (3 levels) in <top (required)>'
# ./lib/gitlab/application_context.rb:59:in `with_raw_context'
# ./spec/spec_helper.rb:401:in `block (2 levels) in <top (required)>'
# ./spec/spec_helper.rb:241:in `block (2 levels) in <top (required)>'
# ./spec/support/system_exit_detected.rb:7:in `block (2 levels) in <main>'
# ./spec/support/redis.rb:17:in `block (3 levels) in <main>'
# ./spec/support/flaky_tests.rb:27:in `block (2 levels) in <main>'
# ./spec/support/database/prevent_cross_joins.rb:106:in `block (3 levels) in <main>'
# ./spec/support/database/prevent_cross_joins.rb:60:in `with_cross_joins_prevented'
# ./spec/support/database/prevent_cross_joins.rb:106:in `block (2 levels) in <main>'
3) ContainerRegistry::Migration::EnqueuerWorker#perform when multiple qualified repositories are available when the deadline is hit does not handle the second qualified repository
Got 2 failures:
3.1) Failure/Error: expect(container_repository.reload).to be_pre_importing
expected `#<ContainerRepository id: 76, project_id: 704, name: "test_image_1", created_at: "2022-11-29 02:11:30...ion_plan: nil, last_cleanup_deleted_tags_count: nil, delete_started_at: nil, status_updated_at: nil>.pre_importing?` to be truthy, got false
# ./spec/workers/container_registry/migration/enqueuer_worker_spec.rb:220:in `block (5 levels) in <main>'
# ./spec/spec_helper.rb:413:in `block (3 levels) in <top (required)>'
# ./spec/support/sidekiq_middleware.rb:18:in `with_sidekiq_server_middleware'
# ./spec/spec_helper.rb:405:in `block (2 levels) in <top (required)>'
# ./spec/spec_helper.rb:401:in `block (3 levels) in <top (required)>'
# ./lib/gitlab/application_context.rb:59:in `with_raw_context'
# ./spec/spec_helper.rb:401:in `block (2 levels) in <top (required)>'
# ./spec/spec_helper.rb:241:in `block (2 levels) in <top (required)>'
# ./spec/support/system_exit_detected.rb:7:in `block (2 levels) in <main>'
# ./spec/support/redis.rb:17:in `block (3 levels) in <main>'
# ./spec/support/flaky_tests.rb:27:in `block (2 levels) in <main>'
# ./spec/support/database/prevent_cross_joins.rb:106:in `block (3 levels) in <main>'
# ./spec/support/database/prevent_cross_joins.rb:60:in `with_cross_joins_prevented'
# ./spec/support/database/prevent_cross_joins.rb:106:in `block (2 levels) in <main>'
3.2) Failure/Error: expect(container_repository2.reload).to be_default
expected `#<ContainerRepository id: 81, project_id: 709, name: "test_image_6", created_at: "2022-11-29 02:11:32...ion_plan: nil, last_cleanup_deleted_tags_count: nil, delete_started_at: nil, status_updated_at: nil>.default?` to be truthy, got false
# ./spec/workers/container_registry/migration/enqueuer_worker_spec.rb:221:in `block (5 levels) in <main>'
# ./spec/spec_helper.rb:413:in `block (3 levels) in <top (required)>'
# ./spec/support/sidekiq_middleware.rb:18:in `with_sidekiq_server_middleware'
# ./spec/spec_helper.rb:405:in `block (2 levels) in <top (required)>'
# ./spec/spec_helper.rb:401:in `block (3 levels) in <top (required)>'
# ./lib/gitlab/application_context.rb:59:in `with_raw_context'
# ./spec/spec_helper.rb:401:in `block (2 levels) in <top (required)>'
# ./spec/spec_helper.rb:241:in `block (2 levels) in <top (required)>'
# ./spec/support/system_exit_detected.rb:7:in `block (2 levels) in <main>'
# ./spec/support/redis.rb:17:in `block (3 levels) in <main>'
# ./spec/support/flaky_tests.rb:27:in `block (2 levels) in <main>'
# ./spec/support/database/prevent_cross_joins.rb:106:in `block (3 levels) in <main>'
# ./spec/support/database/prevent_cross_joins.rb:60:in `with_cross_joins_prevented'
# ./spec/support/database/prevent_cross_joins.rb:106:in `block (2 levels) in <main>'
Finished in 8.27 seconds (files took 34.24 seconds to load)
3 examples, 3 failures
Failed examples:
rspec ./spec/workers/container_registry/migration/enqueuer_worker_spec.rb:101 # ContainerRegistry::Migration::EnqueuerWorker#perform when multiple aborted imports are available when an error occurs does abort that migration
rspec ./spec/workers/container_registry/migration/enqueuer_worker_spec.rb:182 # ContainerRegistry::Migration::EnqueuerWorker#perform when multiple qualified repositories are available when the new pre-import maxes out the capacity starts the pre-import only for one qualified repository
rspec ./spec/workers/container_registry/migration/enqueuer_worker_spec.rb:214 # ContainerRegistry::Migration::EnqueuerWorker#perform when multiple qualified repositories are available when the deadline is hit does not handle the second qualified repository
Might be related to #362932 (closed)
Edited by Jennifer Li