Skip to content

Fix a flaky shared example to allow any event tracked before assertion

Rémy Coutable requested to merge fix-flaky-event-tracking-test into master

What does this MR do and why?

This test is the second most flaky one, as reported in https://app.periscopedata.com/app/gitlab/888968/EP---Flaky-tests?widget=12112039&udv=1474231.

The test only passes on the second try. I believe this is because an unique_active_user event is tracked on first try and not on second.

Allowing any event to be tracked before asserting on the event we want to check resolves this flakiness.

How to set up and validate locally

Without this change

$ bin/rspec -f doc 'spec/requests/api/terraform/state_spec.rb[1:1:1:2:1:1]'
[TEST PROF INFO] Spring detected
Running via Spring preloader in process 79918
Run options: include {:focus=>true, :ids=>{"./spec/requests/api/terraform/state_spec.rb"=>["1:1:1:2:1:1"]}}
==> Starting Gitaly (4a6eb3bfb6ecd8324a39089cb3b59e282936331c) set up...
==> Gitaly set up in 102.262767 seconds...

Test environment set up in 112.773682 seconds

API::Terraform::State
  GET /projects/:id/terraform/state/:name
    behaves like endpoint with unique user tracking
      with maintainer permissions
        behaves like tracking unique hll events
          tracks unique event (FAILED - 1)

Failures:

  1) API::Terraform::State GET /projects/:id/terraform/state/:name behaves like endpoint with unique user tracking with maintainer permissions behaves like tracking unique hll events tracks unique event
     Failure/Error: Gitlab::UsageDataCounters::HLLRedisCounter.track_event('unique_active_user', values: user.id)
     
       Gitlab::UsageDataCounters::HLLRedisCounter received :track_event with unexpected arguments
         expected: ("p_terraform_state_api_unique_users", {:values=>an_instance_of(Integer)})
              got: ("unique_active_user", {:values=>3})
       Diff:
       @@ -1 +1 @@
       -["p_terraform_state_api_unique_users", {:values=>an_instance_of(Integer)}]
       +["unique_active_user", {:values=>3}]
     Shared Example Group: "tracking unique hll events" called from ./spec/requests/api/terraform/state_spec.rb:47
     Shared Example Group: "endpoint with unique user tracking" called from ./spec/requests/api/terraform/state_spec.rb:84
     # ./app/services/users/activity_service.rb:42:in `record_activity'
     # ./app/services/users/activity_service.rb:22:in `block in execute'
     [...]
     # ./spec/requests/api/terraform/state_spec.rb:82:in `block (3 levels) in <main>'
     # ./spec/support/shared_examples/controllers/unique_hll_events_examples.rb:16:in `block (2 levels) in <main>'
     # ./spec/spec_helper.rb:464:in `block (3 levels) in <main>'
     # ./spec/support/sidekiq_middleware.rb:18:in `with_sidekiq_server_middleware'
     # ./spec/spec_helper.rb:456:in `block (2 levels) in <main>'
     # ./spec/spec_helper.rb:452:in `block (3 levels) in <main>'
     # ./lib/gitlab/application_context.rb:61:in `with_raw_context'
     # ./spec/spec_helper.rb:452:in `block (2 levels) in <main>'
     # ./spec/spec_helper.rb:240:in `block (2 levels) in <main>'
     # ./spec/support/system_exit_detected.rb:7:in `block (2 levels) in <main>'
     # ./spec/support/database/prevent_cross_joins.rb:108:in `block (3 levels) in <main>'
     # ./spec/support/database/prevent_cross_joins.rb:62:in `with_cross_joins_prevented'
     # ./spec/support/database/prevent_cross_joins.rb:108:in `block (2 levels) in <main>'
     # -e:1:in `<main>'

Finished in 2 minutes 2.2 seconds (files took 42.24 seconds to load)
1 example, 1 failure

Failed examples:

rspec './spec/requests/api/terraform/state_spec.rb[1:1:1:2:1:1]' # API::Terraform::State GET /projects/:id/terraform/state/:name behaves like endpoint with unique user tracking with maintainer permissions behaves like tracking unique hll events tracks unique event

$ RETRIES=1 bin/rspec -f doc 'spec/requests/api/terraform/state_spec.rb[1:1:1:2:1:1]'
Running via Spring preloader in process 89907
Run options: include {:focus=>true, :ids=>{"./spec/requests/api/terraform/state_spec.rb"=>["1:1:1:2:1:1"]}}

Test environment set up in 10.312533 seconds

API::Terraform::State
  GET /projects/:id/terraform/state/:name
    behaves like endpoint with unique user tracking
      with maintainer permissions
        behaves like tracking unique hll events
          tracks unique event
            
1st Try error in ./spec/support/shared_examples/controllers/unique_hll_events_examples.rb:9:
Gitlab::UsageDataCounters::HLLRedisCounter received :track_event with unexpected arguments
  expected: ("p_terraform_state_api_unique_users", {:values=>an_instance_of(Integer)})
       got: ("unique_active_user", {:values=>8})
Diff:
@@ -1 +1 @@
-["p_terraform_state_api_unique_users", {:values=>an_instance_of(Integer)}]
+["unique_active_user", {:values=>8}]

            
RSpec::Retry: 2nd try ./spec/support/shared_examples/controllers/unique_hll_events_examples.rb:9

Finished in 20.13 seconds (files took 4.87 seconds to load)
1 example, 0 failures

With this change

➜ ~/Code/GitLab/gdk/gitlab git:(fix-flaky-graphql-test)[gcp: gitlab-qa-resources | k8s: qa-resources/default] $ bin/rspec -f doc 'spec/requests/api/terraform/state_spec.rb[1:1:1:2:1:1]'          
Running via Spring preloader in process 3304
Run options: include {:focus=>true, :ids=>{"./spec/requests/api/terraform/state_spec.rb"=>["1:1:1:2:1:1"]}}

Test environment set up in 9.248093 seconds

API::Terraform::State
  GET /projects/:id/terraform/state/:name
    behaves like endpoint with unique user tracking
      with maintainer permissions
        behaves like tracking unique hll events
          tracks unique event

Finished in 19.03 seconds (files took 4.45 seconds to load)
1 example, 0 failures

MR acceptance checklist

This checklist encourages us to confirm any changes have been analyzed to reduce risks in quality, performance, reliability, security, and maintainability.

Edited by Rémy Coutable

Merge request reports