Skip to content

Run RSpec bisect for failed specs during order dependenance check

Peter Leitzenrequested to merge
pl-check-order-dependence-improvements into master

What does this MR do and why?

When scripts/rspec_check_order_dependence detects a failure it'd be great to run rspec --bisect to create a minimal reproduction.

This MR adds this functionality!

Note that if rspec --bisect is run, the script exits and does not remove passed specs from spec/support/rspec_order_todo.yml yet.

References

How to set up and validate locally

  1. Run scripts/rspec_check_order_dependence spec/lib/gitlab/import*
  2. See minimal reproduction:
Round 1: bisecting over non-failing examples 1-69
. ignoring examples 1-35 (18.53 seconds)
Round 2: bisecting over non-failing examples 36-69
. ignoring examples 36-52 (13 seconds)
Round 3: bisecting over non-failing examples 53-69
. ignoring examples 53-61 (8.74 seconds)
Round 4: bisecting over non-failing examples 62-69
. ignoring examples 62-65 (8.55 seconds)
Round 5: bisecting over non-failing examples 66-69
. ignoring examples 66-67 (8.49 seconds)
Round 6: bisecting over non-failing examples 68-69
. ignoring example 68 (7.82 seconds)
Bisect complete! Reduced necessary non-failing examples from 69 to 1 in 1 minute 12.54 seconds.

The minimal reproduction command is:
  rspec ./spec/lib/gitlab/import/page_counter_spec.rb[1:2:1,1:2:2] --fail-fast --format progress --tag ~quarantine -rspec_helper --seed 57375 --order defined
Click to expand 
NOTE: Specs are mixing fast spec helpers with other spec helpers. Adding -rspec_helper.
+ export RSPEC_WARN_MISSING_FEATURE_CATEGORY=0
+ RSPEC_WARN_MISSING_FEATURE_CATEGORY=0
+ echo 'Running in defined order:'
Running in defined order:
+ rspec_run --order defined spec/lib/gitlab/import spec/lib/gitlab/import_export spec/lib/gitlab/import_formatter_spec.rb spec/lib/gitlab/import_sources_spec.rb
+ bin/rspec --fail-fast --format progress --tag '~quarantine' -rspec_helper --order defined spec/lib/gitlab/import spec/lib/gitlab/import_export spec/lib/gitlab/import_formatter_spec.rb spec/lib/gitlab/import_sources_spec.rb
+ tee /tmp/rspec-check-order_dependence-INS3Do
Run options:
  include {:focus=>true}
  exclude {:quarantine=>true}

All examples were filtered out; ignoring {:focus=>true}
Running via Spring preloader in process 282400

Test environment set up in 0.8608846440038178 seconds
.....................................................................F

Failures:

  1) Gitlab::Import::PageCounter#set does not overwrite the page number when the given number is lower than the current number
     Failure/Error: expect(counter.current).to eq(2)

       expected: 2
            got: 4

       (compared using ==)
     # ./spec/lib/gitlab/import/page_counter_spec.rb:41:in `block (3 levels) in <main>'
     # ./spec/spec_helper.rb:501:in `block (3 levels) in <main>'
     # ./lib/gitlab/sidekiq_sharding/validator.rb:42:in `enabled'
     # ./spec/spec_helper.rb:500:in `block (2 levels) in <main>'
     # ./spec/spec_helper.rb:495:in `block (3 levels) in <main>'
     # ./spec/support/sidekiq_middleware.rb:9:in `with_sidekiq_server_middleware'
     # ./spec/spec_helper.rb:486:in `block (2 levels) in <main>'
     # ./spec/spec_helper.rb:482:in `block (3 levels) in <main>'
     # ./lib/gitlab/application_context.rb:102:in `with_raw_context'
     # ./spec/spec_helper.rb:482:in `block (2 levels) in <main>'
     # ./spec/spec_helper.rb:453:in `block (3 levels) in <main>'
     # ./lib/gitlab/ci/config/feature_flags.rb:38:in `ensure_correct_usage'
     # ./spec/spec_helper.rb:452:in `block (2 levels) in <main>'
     # ./spec/support/system_exit_detected.rb:7:in `block (2 levels) in <main>'
     # ./spec/support/redis.rb:18:in `block (3 levels) in <main>'
     # ./spec/support/database/prevent_cross_joins.rb:106:in `block (3 levels) in <main>'
     # ./spec/support/database/prevent_cross_joins.rb:60:in `with_cross_joins_prevented'
     # ./spec/support/database/prevent_cross_joins.rb:106:in `block (2 levels) in <main>'
     # -e:1:in `<main>'

Finished in 23.15 seconds (files took 6.95 seconds to load)
70 examples, 1 failure

Failed examples:

rspec ./spec/lib/gitlab/import/page_counter_spec.rb:37 # Gitlab::Import::PageCounter#set does not overwrite the page number when the given number is lower than the current number

Randomized with seed 57375

[TEST PROF INFO] Time spent in factories: 00:09.690 (32.31% of total time)
+ echo 'WARNING: Specs failed. Running RSpec bisect:'
WARNING: Specs failed. Running RSpec bisect:
+ rspec_bisect --order defined spec/lib/gitlab/import spec/lib/gitlab/import_export spec/lib/gitlab/import_formatter_spec.rb spec/lib/gitlab/import_sources_spec.rb
++ grep 'Randomized with seed ' /tmp/rspec-check-order_dependence-INS3Do
++ grep -oE '[0-9]+'
++ head -1
+ seed=57375
+ local args
+ args=("${RSPEC_ARGS[@]}")
+ [[ -n 57375 ]]
+ echo 'NOTE: Found seed 57375. Adding --seed 57375.'
NOTE: Found seed 57375. Adding --seed 57375.
+ args+=(--seed "$seed")
+ bin/rspec --bisect --fail-fast --format progress --tag '~quarantine' -rspec_helper --seed 57375 --order defined spec/lib/gitlab/import spec/lib/gitlab/import_export spec/lib/gitlab/import_formatter_spec.rb spec/lib/gitlab/import_sources_spec.rb

[TEST PROF INFO] Spring detected

Running via Spring preloader in process 282867
Bisect started using options: "--fail-fast --format progress --tag ~quarantine -rspec_helper --seed 57375 --order defined spec/lib/gitlab/import spec/lib/gitlab/import_export spec/lib/gitlab/import_formatter_spec.rb spec/lib/gitlab/import_sources_spec.rb"
Running suite to find failures... (23.98 seconds)
Starting bisect with 1 failing example and 69 non-failing examples.
Checking that failure(s) are order-dependent..[TEST PROF INFO] Time spent in factories: 00:08.867 (29.96% of total time)
. failure appears to be order-dependent

Round 1: bisecting over non-failing examples 1-69 [TEST PROF INFO] Time spent in factories: 00:00.056 (0.15% of total time)
. ignoring examples 1-35 (18.53 seconds)
Round 2: bisecting over non-failing examples 36-69 [TEST PROF INFO] Time spent in factories: 00:07.065 (12.72% of total time)
. ignoring examples 36-52 (13 seconds)
Round 3: bisecting over non-failing examples 53-69 [TEST PROF INFO] Time spent in factories: 00:04.342 (6.34% of total time)
. ignoring examples 53-61 (8.74 seconds)
Round 4: bisecting over non-failing examples 62-69 [TEST PROF INFO] Time spent in factories: 00:00.208 (0.27% of total time)
. ignoring examples 62-65 (8.55 seconds)
Round 5: bisecting over non-failing examples 66-69 [TEST PROF INFO] Time spent in factories: 00:00.048 (0.06% of total time)
. ignoring examples 66-67 (8.49 seconds)
Round 6: bisecting over non-failing examples 68-69 [TEST PROF INFO] Time spent in factories: 00:00.032 (0.03% of total time)
. ignoring example 68 (7.82 seconds)
Bisect complete! Reduced necessary non-failing examples from 69 to 1 in 1 minute 12.54 seconds.

The minimal reproduction command is:
  rspec ./spec/lib/gitlab/import/page_counter_spec.rb[1:2:1,1:2:2] --fail-fast --format progress --tag ~quarantine -rspec_helper --seed 57375 --order defined

[TEST PROF INFO] Time spent in factories: 00:00.034 (0.03% of total time)
+ exit 1
+ cleanup
+ rm -f /tmp/rspec-check-order_dependence-INS3Do

MR acceptance checklist

Evaluate this MR against the MR acceptance checklist. It helps you analyze changes to reduce risks in quality, performance, reliability, security, and maintainability.

Edited by Peter Leitzen

Merge request reports