Skip to content

Fix flaky vault_spec

Peter Leitzen requested to merge pl-spec-flaky-spec-gdk-root into main

What does this merge request do and why?

It seems that using real GDK root and accessing GDK.config doesn't play well and cause flaky specs.

The reason is that GDK.config is cached which includes the unstubbed GDK root (config.gdk_root).

Ideally, we would pass GDK root explicitly to config.

However, in this case :gdk_root is not even needed as all file access is stubbed as well.

Refs #2125

How to set up and validate locally

# On main
bundle exec rspec --format documentation --order defined spec/lib/gdk/announcements_spec.rb spec/gdk/services/vault_spec.rb

GDK::Announcements
  #render_all
    renders an announcement to the terminal
    with multiple announcements
      renders all announcements to the terminal

GDK::Services::Vault
  #name
    returns vault
  #command
    returns command based on config (FAILED - 1)
  #enabled?
    returns true if set `enabled: true` in the config file
    returns false if set `enabled: false` in the config file

Failures:

  1) GDK::Services::Vault#command returns command based on config
     Failure/Error: expect(subject.command).to match(/vault server --dev --dev-listen-address=127.0.0.1:8200/)

       expected "/usr/local/bin/vault server --dev --dev-listen-address=10.23.0.5:8200" to match /vault server --dev --dev-listen-address=127.0.0.1:8200/
       Diff:
       @@ -1 +1 @@
       -/vault server --dev --dev-listen-address=127.0.0.1:8200/
       +"/usr/local/bin/vault server --dev --dev-listen-address=10.23.0.5:8200"

     # ./spec/gdk/services/vault_spec.rb:12:in `block (3 levels) in <top (required)>'
     # spec/spec_support/system_exit_detected.rb:7:in `block (2 levels) in <top (required)>'

Finished in 0.0584 seconds (files took 0.21079 seconds to load)
6 examples, 1 failure

Failed examples:

rspec ./spec/gdk/services/vault_spec.rb:11 # GDK::Services::Vault#command returns command based on config

Coverage report generated for RSpec to /home/peter/devel/gitlab/gdk/coverage. 508 / 1406 LOC (36.13%) covered.
Coverage report generated for RSpec to /home/peter/devel/gitlab/gdk/coverage/coverage.xml. 508 / 1406 LOC (36.13%) covered
Stopped processing SimpleCov as a previous error not related to SimpleCov has been detected

Found via RSpec bisect:

bundle exec rspec --bisect --seed 61856
Bisect started using options: "--seed 61856"
Running suite to find failures... (6.74 seconds)
Starting bisect with 5 failing examples and 1284 non-failing examples.
Checking that failure(s) are order-dependent... failure appears to be order-dependent

Round 1: bisecting over non-failing examples 1-1284 .. ignoring examples 643-1284 (0.79865 seconds)
Round 2: bisecting over non-failing examples 1-642 .. ignoring examples 322-642 (0.72283 seconds)
Round 3: bisecting over non-failing examples 1-321 .. ignoring examples 162-321 (0.59895 seconds)
Round 4: bisecting over non-failing examples 1-161 .. ignoring examples 82-161 (0.42429 seconds)
Round 5: bisecting over non-failing examples 1-81 .. ignoring examples 42-81 (0.30834 seconds)
Round 6: bisecting over non-failing examples 1-41 .. ignoring examples 22-41 (0.24826 seconds)
Round 7: bisecting over non-failing examples 1-21 .. ignoring examples 12-21 (0.22735 seconds)
Round 8: bisecting over non-failing examples 1-11 .. ignoring examples 7-11 (0.19596 seconds)
Round 9: bisecting over non-failing examples 1-6 .. ignoring examples 4-6 (0.19477 seconds)
Round 10: bisecting over non-failing examples 1-3 .. ignoring example 3 (0.17767 seconds)
Round 11: bisecting over non-failing examples 1-2 . ignoring example 1 (0.09199 seconds)
Bisect complete! Reduced necessary non-failing examples from 1284 to 1 in 4.07 seconds.

The minimal reproduction command is:
  rspec ./spec/gdk/services/vault_spec.rb[1:2:1] ./spec/lib/gdk/announcements_spec.rb[1:1:2:1] ./spec/lib/gdk/services/gitlab_workhorse_spec.rb[1:2:1] ./spec/lib/gdk/services/minio_spec.rb[1:2:1] ./spec/lib/gdk/services/redis_cluster_spec.rb[1:2:1] ./spec/lib/gdk/services/vault_spec.rb[1:2:1] --seed 61856

Coverage report generated for RSpec to /home/peter/devel/gitlab/gdk/coverage. 23 / 33 LOC (69.7%) covered.
Coverage report generated for RSpec to /home/peter/devel/gitlab/gdk/coverage/coverage.xml. 23 / 33 LOC (69.70%) covered

All specs pass on this branch:

bundle exec rspec ./spec/gdk/services/vault_spec.rb[1:2:1] ./spec/lib/gdk/announcements_spec.rb[1:1:2:1] ./spec/lib/gdk/services/gitlab_workhorse_spec.rb[1:2:1] ./spec/lib/gdk/services/minio_spec.rb[1:2:1] ./spec/lib/gdk/services/redis_cluster_spec.rb[1:2:1] ./spec/lib/gdk/services/vault_spec.rb[1:2:1] --seed 61856
Run options: include {:ids=>{"./spec/gdk/services/vault_spec.rb"=>["1:2:1"], "./spec/lib/gdk/announcements_spec.rb"=>["1:1:2:1"], "./spec/lib/gdk/services/gitlab_workhorse_spec.rb"=>["1:2:1"], "./spec/lib/gdk/services/minio_spec.rb"=>["1:2:1"], "./spec/lib/gdk/services/redis_cluster_spec.rb"=>["1:2:1"], "./spec/lib/gdk/services/vault_spec.rb"=>["1:2:1"]}}

Randomized with seed 61856
......

Finished in 0.05516 seconds (files took 0.21513 seconds to load)
6 examples, 0 failures

Randomized with seed 61856

Coverage report generated for RSpec to /home/peter/devel/gitlab/gdk/coverage. 702 / 1553 LOC (45.2%) covered.
Coverage report generated for RSpec to /home/peter/devel/gitlab/gdk/coverage/coverage.xml. 702 / 1553 LOC (45.20%) covered

Impacted categories

The following categories relate to this merge request:

Merge request checklist

  • This change is backward compatible. If not, please include steps to communicate to our users.
  • Tests added for new functionality. If not, please raise an issue to follow-up.
  • Documentation added/updated, if needed.
  • Announcement added, if change is notable.
  • gdk doctor test added, if needed.
  • Add the ~highlight label if this MR should be included in the CHANGELOG.md.
Edited by Peter Leitzen

Merge request reports