Failure in qa/specs/features/ee/browser_ui/4_verify/cancelling_merge_request_in_merge_train_spec.rb | @project.add_member RetriesExceededError

Summary

Test fails with adding a project member as a Maintainer in the before block.

Job:

• https://ops.gitlab.net/gitlab-org/quality/staging/-/jobs/4866738

Stack trace

Failures:
  1) Verify Cancelling merge request in merge train when user cancels the merge request does not create a TODO task
     Failure/Error: @project.add_member(@user, Resource::Members::AccessLevel::MAINTAINER)
     
     QA::Support::Repeater::RetriesExceededError:
       Retry condition not met after 3 attempts
     # ./qa/support/repeater.rb:48:in `repeat_until'
     # ./qa/support/retrier.rb:50:in `retry_until'
     # ./qa/resource/members.rb:11:in `add_member'
     # ./qa/specs/features/ee/browser_ui/4_verify/cancelling_merge_request_in_merge_train_spec.rb:65:in `block (3 levels) in <module:QA>'
     # /usr/local/bundle/gems/rspec-retry-0.6.1/lib/rspec/retry.rb:123:in `block in run'
     # /usr/local/bundle/gems/rspec-retry-0.6.1/lib/rspec/retry.rb:110:in `loop'
     # /usr/local/bundle/gems/rspec-retry-0.6.1/lib/rspec/retry.rb:110:in `run'
     # /usr/local/bundle/gems/rspec-retry-0.6.1/lib/rspec_ext/rspec_ext.rb:12:in `run_with_retry'
     # ./spec/spec_helper.rb:81:in `block (2 levels) in <top (required)>'
     # /usr/local/bundle/gems/rspec-retry-0.6.1/lib/rspec/retry.rb:123:in `block in run'
     # /usr/local/bundle/gems/rspec-retry-0.6.1/lib/rspec/retry.rb:110:in `loop'
     # /usr/local/bundle/gems/rspec-retry-0.6.1/lib/rspec/retry.rb:110:in `run'
     # /usr/local/bundle/gems/rspec-retry-0.6.1/lib/rspec_ext/rspec_ext.rb:12:in `run_with_retry'
     # /usr/local/bundle/gems/rspec-retry-0.6.1/lib/rspec/retry.rb:37:in `block (2 levels) in setup'
     # ./qa/specs/runner.rb:74:in `perform'
     # ./qa/scenario/template.rb:10:in `block in perform'
     # ./qa/scenario/template.rb:8:in `tap'
     # ./qa/scenario/template.rb:8:in `perform'
     # ./qa/scenario/template.rb:44:in `perform'
     # ./qa/scenario/template.rb:10:in `block in perform'
     # ./qa/scenario/template.rb:8:in `tap'
     # ./qa/scenario/template.rb:8:in `perform'
     # ./qa/scenario/bootable.rb:28:in `launch!'

Screenshot / HTML page

Indicates previous page after enabling merge trains. Not useful.

Possible fixes

changed due date to September 28, 2021

changed milestone to %14.4

added QA Quality devopsverify failureinvestigating found:staging.gitlab.com priority1 test labels

mentioned in issue gitlab-org/quality/pipeline-triage#99 (closed)

added sectionops label

added [deprecated] Accepting merge requests label

Came up again on Staging: https://ops.gitlab.net/gitlab-org/quality/staging/-/jobs/4900948

mentioned in issue gitlab-org/quality/pipeline-triage#100 (closed)

mentioned in issue gitlab-org/quality/triage-reports#4659 (closed)

added Category:Merge Trains label

Adding Merge Trains

Setting label(s) grouppipeline execution based on Category:Merge Trains.

added grouppipeline execution label

Another failure on staging - https://ops.gitlab.net/gitlab-org/quality/staging/-/jobs/4928497

added failureflaky-test label and removed Category:Merge Trains QA Quality [deprecated] Accepting merge requests devopsverify failureinvestigating found:staging.gitlab.com grouppipeline execution priority1 sectionops test labels

added Category:Merge Trains QA Quality [deprecated] Accepting merge requests devopsverify found:staging.gitlab.com grouppipeline execution priority2 sectionops test labels

mentioned in issue gitlab-org/quality/pipeline-triage#101 (closed)

mentioned in issue gitlab-org/quality/pipeline-triage#102 (closed)

Staging https://ops.gitlab.net/gitlab-org/quality/staging/-/jobs/5088812

Staging https://ops.gitlab.net/gitlab-org/quality/staging/-/jobs/5101605

changed the description

Seen on Staging https://ops.gitlab.net/gitlab-org/quality/staging/-/jobs/5125987

Very odd ... Where the test failed, it was supposed to be on Project's Settings page, if the page was indeed hit 404, then the test should have failed at trying to enable merge trains steps But it completed the steps and moved on to create user and add user via API ... I couldn't reproduce locally so I still couldn't understand the root cause

Hm ... took a closer look, according to pipeline log https://ops.gitlab.net/gitlab-org/quality/staging/-/jobs/5125987#L1239 ... Test tried 3 times but couldn't add user to project.

mentioned in issue gitlab-org/quality/pipeline-triage#103 (closed)

assigned to @treagitlab

removed [deprecated] Accepting merge requests label

changed milestone to %14.5

added missed:14.4 label

Again on Staging https://ops.gitlab.net/gitlab-org/quality/staging/-/jobs/5225370 . I wonder if this issue and #343315 (closed) are related?

mentioned in issue #343315 (closed)

mentioned in issue gitlab-org/quality/pipeline-triage#104 (closed)

I've been running this spec against Staging multiple times last week and this failure didn't come up at all.. At the same time, this failure is happening across multiple specs too. I'll keep monitoring for now. 🧐

I'll move this to the backlog for now to continue observing for now.

changed milestone to %Backlog

unassigned @treagitlab

Again on Staging https://ops.gitlab.net/gitlab-org/quality/staging/-/jobs/5515684 but 500 this time

Verify Cancelling merge request in merge train when user cancels the merge request does not create a TODO task
     Failure/Error: @project.add_member(@user, Resource::Members::AccessLevel::MAINTAINER)
     
     QA::Support::Repeater::RetriesExceededError:
       Retry failed after 3 attempts

I went through Sentry and found something that's possibly related (https://sentry.gitlab.net/gitlab/staginggitlabcom/issues/3020795). None of the other errors that occurred in the past 3 hours has anything closer too.

ActionView::Template::ErrorProjectsController#show
  unable to nest the invite_members_in_side_nav experiment within the invite_members_in_side_nav experiment

There was an occurrence around 8:48 PM UTC which is shortly before this build failed and this is related to adding members.

I'll look into it more closely to understand what this experiment is doing and what might have caused this issue.

Another one in Staging https://ops.gitlab.net/gitlab-org/quality/staging/-/jobs/5516538 but slightly different

We are inviting this member via API, I'm not sure what side nav has anything to do here?

Oh oops! I didn't see this until just now. You're right it should be through API so the experiment with the side nav shouldn't have anything to do with it.. I'll keep looking.

mentioned in issue gitlab-org/quality/pipeline-triage#109 (closed)

added [deprecated] Accepting merge requests label

Seeing this in Staging with the 500 error screenshot

cc @richard.chong

https://ops.gitlab.net/gitlab-org/quality/staging/-/jobs/5610132

mentioned in issue gitlab-org/quality/pipeline-triage#110 (closed)

mentioned in issue gitlab-org/quality/pipeline-triage#111 (closed)

Seeing this causing failing examples in master
Failure/Error: Flow::MergeRequest.enable_merge_trains

• Verify Cancelling merge request in merge train when system cancels the merge request creates a TODO task
• Verify Cancelling merge request in merge train when user cancels the merge request does not create a TODO task

mentioned in issue gitlab-org/quality/pipeline-triage#113 (closed)

mentioned in issue #346047 (closed)

Seen on Staging

• https://ops.gitlab.net/gitlab-org/quality/staging/-/jobs/6201041
• https://ops.gitlab.net/gitlab-org/quality/staging/-/jobs/6201787

Note, this is a triggered pipeline when FF create_project_namespace_on_project_create is turned on. Not that I'm saying this FF is related to this failure, but more of perhaps when something new is kicked start, it might cause some infra hiccups? Worth keeping in mind.

Well, this is now failing consistently in Staging full run so we ended up quarantining the test !79785 (merged)

Failed in staging :https://ops.gitlab.net/gitlab-org/quality/staging/-/jobs/6215376 waiting for quarantine MR to be merged

I've been trying to reproduce this running it against Staging and I can't reproduce this failure. I tried to flag it down in Kibana to see what the 404 is about but unfortunately the logs for these ones are gone now.

I'm going to unquarantine this just in time for next week when I'm DRI for triage so I can monitor it closely. I'll hopefully be able to flag down what caused the 404 through Kibana if it does come up again.

mentioned in merge request !79785 (merged)

mentioned in issue gitlab-org/quality/pipeline-triage#125 (closed)

mentioned in merge request !80461 (merged)

mentioned in issue gitlab-org/quality/pipeline-triage#127 (closed)

mentioned in issue gitlab-org/quality/pipeline-triage#128 (closed)

mentioned in issue gitlab-org/quality/pipeline-triage#129 (closed)

removed due date

Failure occurred in staging pipeline: https://ops.gitlab.net/gitlab-org/quality/staging/-/jobs/6448236

added Enterprise Edition label

mentioned in issue gitlab-org/quality/testcase-sessions#39831 (closed)

Failure occurred in staging pipeline: https://ops.gitlab.net/gitlab-org/quality/staging/-/jobs/6451222

mentioned in issue gitlab-org/quality/testcase-sessions#39869 (closed)

Failure occurred in staging pipeline: https://ops.gitlab.net/gitlab-org/quality/staging/-/jobs/6456460

mentioned in issue gitlab-org/quality/testcase-sessions#39931 (closed)

mentioned in issue gitlab-org/ci-cd/pipeline-execution#88 (closed)

changed milestone to %14.10

Failure occurred in staging pipeline: https://ops.gitlab.net/gitlab-org/quality/staging/-/jobs/6462120

mentioned in issue gitlab-org/quality/testcase-sessions#39980 (closed)

mentioned in issue gitlab-org/quality/testcase-sessions#39981 (closed)

added needs weight workflowplanning breakdown labels

From the log, it looks like there might have been a delay with creating the merge request via API which caused it to hit a 404 error when trying to visit it. I'll introduce a check to ensure that the merge request exists first before visiting it.

@richard.chong could you put a weight estimate on this issue and if you're taking it assign yourself? Thanks!

Failure occurred in staging pipeline: https://ops.gitlab.net/gitlab-org/quality/staging/-/jobs/6492827

mentioned in issue gitlab-org/quality/testcase-sessions#40296 (closed)

Failure occurred in staging pipeline: https://ops.gitlab.net/gitlab-org/quality/staging/-/jobs/6495004

mentioned in issue gitlab-org/quality/testcase-sessions#40322 (closed)

mentioned in merge request !82452 (merged)

assigned to @richard.chong

set weight to 1

changed milestone to %14.9

added workflowin dev label and removed workflowplanning breakdown label

Failure occurred in staging pipeline: https://ops.gitlab.net/gitlab-org/quality/staging/-/jobs/6512269

mentioned in issue gitlab-org/quality/testcase-sessions#40514 (closed)

Thanks for working on this @richard.chong! We've removed the Seeking community contributions label to avoid having multiple people working on the same issue.

removed [deprecated] Accepting merge requests label

MR with a fix has been merged !82452 (merged)

closed

added workflowproduction label and removed workflowin dev label