2023-05-04: QA failures on gstg-cny blocking deployments
Customer Impact
Current Status
More information will be added as we investigate the issue. For customers believed to be affected by this incident, please subscribe to this issue or monitor our status page for further updates.

References and helpful links

Recent Events (available internally only):
- Feature Flag Log - Chatops to toggle Feature Flags Documentation
- Infrastructure Configurations
- GCP Events (e.g. host failure)
Deployment Guidance
- Deployments Log | Gitlab.com Latest Updates
- Reach out to Release Managers for S1/S2 incidents to discuss Rollbacks, Hot Patching or speeding up deployments. | Rollback Runbook | Hot Patch Runbook
Use the following links to create related issues to this incident if additional work needs to be completed after it is resolved:
- Corrective action ❙ Infradev
- Incident Review ❙ Infra investigation followup
- Confidential Support contact ❙ QA investigation
Note: In some cases we need to redact information from public view. We only do this in a limited number of documented cases. This might include the summary, timeline or any other bits of information, laid out in out handbook page. Any of this confidential data will be in a linked issue, only visible internally. By default, all information we can share, will be public, in accordance to our transparency value.
No timeline items have been added yet.
- Show closed items
Activity
-
Newest first Oldest first
-
Show all activity Show comments only Show history only
- ops-gitlab-net added an incident timeline event
added an incident timeline event
added IncidentActive Source::IMAIncidentDeclare incident severity3 labels
- ops-gitlab-net assigned to @msmiley
assigned to @msmiley
- ops-gitlab-net changed the description
Compare with previous version changed the description
- ops-gitlab-net changed the severity to Medium - S3
changed the severity to Medium - S3
- ops-gitlab-net added a resource link
added a resource link
- Owner
There are two qa-reliable tests failing on gstg-cny. This blocks deployments:
Failures: 1) Create Reverting a commit creates a merge request Failure/Error: Page::MergeRequest::New.perform(&:create_merge_request) QA::Page::Validatable::PageValidationError: discussion_preferences_dropdown did not appear on QA::Page::MergeRequest::Show as expected # ./qa/page/validatable.rb:15:in `block in validate_elements_present!' # ./qa/page/validatable.rb:13:in `each' # ./qa/page/validatable.rb:13:in `validate_elements_present!' # ./qa/page/base.rb:234:in `click_element' # ./qa/support/page/logging.rb:87:in `block in click_element' # ./qa/support/page/logging.rb:206:in `log_slow_code' # ./qa/support/page/logging.rb:87:in `click_element' # ./qa/page/merge_request/new.rb:41:in `create_merge_request' # ./qa/scenario/actable.rb:16:in `perform' # ./qa/specs/features/browser_ui/3_create/merge_request/revert/revert_commit_spec.rb:30:in `block (3 levels) in <module:QA>' # ./qa/specs/spec_helper.rb:121:in `block (2 levels) in <top (required)>' # ./qa/specs/knapsack_runner.rb:17:in `run' # ./qa/specs/runner.rb:57:in `perform' # ./qa/scenario/template.rb:10:in `block in perform' # ./qa/scenario/template.rb:8:in `perform' # ./qa/scenario/template.rb:50:in `perform' # ./qa/scenario/template.rb:10:in `block in perform' # ./qa/scenario/template.rb:8:in `perform' # ./qa/scenario/bootable.rb:52:in `launch!' Finished in 9 minutes 26 seconds (files took 2.99 seconds to load) 14 examples, 1 failure /builds/gitlab-org/quality/staging/.gems/bin/gitlab-qa: Command `docker run -t --rm --net=bridge --env QA_CAN_TEST_GIT_PROTOCOL_V2="$QA_CAN_TEST_GIT_PROTOCOL_V2" ...` failed! \e[31m✘\e[0m (Gitlab::QA::Support::ShellCommand::StatusError) Failed examples: rspec ./qa/specs/features/browser_ui/3_create/merge_request/revert/revert_commit_spec.rb:28 # Create Reverting a commit creates a merge request Randomized with seed 3535
Failures: 1) Create Merge request custom templates creates a merge request via custom template Failure/Error: Resource::MergeRequest.fabricate_via_browser_ui! do |merge_request| merge_request.project = template_project merge_request.title = merge_request_title merge_request.template = template_name merge_request.target_new_branch = false end QA::Page::Validatable::PageValidationError: discussion_preferences_dropdown did not appear on QA::Page::MergeRequest::Show as expected # ./qa/page/validatable.rb:15:in `block in validate_elements_present!' # ./qa/page/validatable.rb:13:in `each' # ./qa/page/validatable.rb:13:in `validate_elements_present!' # ./qa/page/base.rb:234:in `click_element' # ./qa/support/page/logging.rb:87:in `block in click_element' # ./qa/support/page/logging.rb:206:in `log_slow_code' # ./qa/support/page/logging.rb:87:in `click_element' # ./qa/page/merge_request/new.rb:41:in `create_merge_request' # ./qa/resource/merge_request.rb:102:in `block in fabricate!' # ./qa/scenario/actable.rb:16:in `perform' # ./qa/resource/merge_request.rb:91:in `fabricate!' # ./qa/resource/base.rb:49:in `block (2 levels) in fabricate_via_browser_ui!' # ./qa/resource/base.rb:96:in `log_and_record_fabrication' # ./qa/resource/base.rb:49:in `block in fabricate_via_browser_ui!' # ./qa/resource/base.rb:86:in `do_fabricate!' # ./qa/resource/base.rb:48:in `fabricate_via_browser_ui!' # ./qa/specs/features/browser_ui/3_create/merge_request/create_merge_request_via_template_spec.rb:34:in `block (3 levels) in <module:QA>' # ./qa/specs/spec_helper.rb:121:in `block (2 levels) in <top (required)>' # ./qa/specs/knapsack_runner.rb:17:in `run' # ./qa/specs/runner.rb:57:in `perform' # ./qa/scenario/template.rb:10:in `block in perform' # ./qa/scenario/template.rb:8:in `perform' # ./qa/scenario/template.rb:50:in `perform' # ./qa/scenario/template.rb:10:in `block in perform' # ./qa/scenario/template.rb:8:in `perform' # ./qa/scenario/bootable.rb:52:in `launch!' Finished in 9 minutes 42 seconds (files took 3.32 seconds to load) 11 examples, 1 failure Failed examples: rspec ./qa/specs/features/browser_ui/3_create/merge_request/create_merge_request_via_template_spec.rb:33 # Create Merge request custom templates creates a merge request via custom template
Failures: 1) Create Merge request custom templates creates a merge request via custom template Failure/Error: Resource::MergeRequest.fabricate_via_browser_ui! do |merge_request| merge_request.project = template_project merge_request.title = merge_request_title merge_request.template = template_name merge_request.target_new_branch = false end QA::Page::Validatable::PageValidationError: discussion_preferences_dropdown did not appear on QA::Page::MergeRequest::Show as expected # ./qa/page/validatable.rb:15:in `block in validate_elements_present!' # ./qa/page/validatable.rb:13:in `each' # ./qa/page/validatable.rb:13:in `validate_elements_present!' # ./qa/page/base.rb:234:in `click_element' # ./qa/support/page/logging.rb:87:in `block in click_element' # ./qa/support/page/logging.rb:206:in `log_slow_code' # ./qa/support/page/logging.rb:87:in `click_element' # ./qa/page/merge_request/new.rb:41:in `create_merge_request' # ./qa/resource/merge_request.rb:102:in `block in fabricate!' # ./qa/scenario/actable.rb:16:in `perform' # ./qa/resource/merge_request.rb:91:in `fabricate!' # ./qa/resource/base.rb:49:in `block (2 levels) in fabricate_via_browser_ui!' # ./qa/resource/base.rb:96:in `log_and_record_fabrication' # ./qa/resource/base.rb:49:in `block in fabricate_via_browser_ui!' # ./qa/resource/base.rb:86:in `do_fabricate!' # ./qa/resource/base.rb:48:in `fabricate_via_browser_ui!' # ./qa/specs/features/browser_ui/3_create/merge_request/create_merge_request_via_template_spec.rb:34:in `block (3 levels) in <module:QA>' # ./qa/specs/spec_helper.rb:121:in `block (2 levels) in <top (required)>' # ./qa/specs/knapsack_runner.rb:17:in `run' # ./qa/specs/runner.rb:57:in `perform' # ./qa/scenario/template.rb:10:in `block in perform' # ./qa/scenario/template.rb:8:in `perform' # ./qa/scenario/template.rb:50:in `perform' # ./qa/scenario/template.rb:10:in `block in perform' # ./qa/scenario/template.rb:8:in `perform' # ./qa/scenario/bootable.rb:52:in `launch!' Finished in 9 minutes 42 seconds (files took 3.32 seconds to load) 11 examples, 1 failure Failed examples: rspec ./qa/specs/features/browser_ui/3_create/merge_request/create_merge_request_via_template_spec.rb:33 # Create Merge request custom templates creates a merge request via custom template
Edited by Steve Abrams - Steve Abrams added Delivery impact2 label
added Delivery impact2 label
- Sofia Vistas mentioned in issue gitlab-org/gitlab#408218 (closed)
mentioned in issue gitlab-org/gitlab#408218 (closed)
- Owner
Feature flag
inherit_approval_rules_on_creation
has been identified as a potential cause. It has been disabled and the jobs have been retried.Edited by Steve Abrams 1 Collapse replies - Owner
The jobs continue to fail, so I will restore the feature flag to it's previous state.
1 - Owner
Now we are going to try disabling
mr_activity_filters
as it was also enabled in the same time period on staging. - Owner
The tests have all succeeded on retry. I'm marking this incident as mitigated and have informed the author of the QA failures.
- Sofia Vistas mentioned in issue gitlab-org/quality/pipeline-triage#196 (closed)
mentioned in issue gitlab-org/quality/pipeline-triage#196 (closed)
- Steve Abrams mentioned in issue gitlab-org/release/tasks#5592 (closed)
mentioned in issue gitlab-org/release/tasks#5592 (closed)
- 🤖 GitLab Bot 🤖 added NeedsRootCause label
added NeedsRootCause label
- 🤖 GitLab Bot 🤖 added NeedsService label
added NeedsService label
- Steve Abrams added IncidentMitigated label and removed IncidentActive label
added IncidentMitigated label and removed IncidentActive label
- Steve Abrams added IncidentResolved label and removed IncidentMitigated label
added IncidentResolved label and removed IncidentMitigated label
- Maintainer
This incident was automatically closed because it has the IncidentResolved label.
Note: All incidents are closed automatically when they are resolved, even when there is a pending review. Please see the Incident Workflow section on the Incident Management handbook page for more information.
- 🤖 GitLab Bot 🤖 closed
closed
- 🤖 GitLab Bot 🤖 changed the incident status to Resolved by closing the incident
changed the incident status to Resolved by closing the incident
- 🤖 GitLab Bot 🤖 removed NeedsRootCause label
removed NeedsRootCause label
- ops-gitlab-net mentioned in issue on-call-handovers#3820 (closed)
mentioned in issue on-call-handovers#3820 (closed)
- ops-gitlab-net mentioned in issue reliability-reports#165 (closed)
mentioned in issue reliability-reports#165 (closed)
- GitLab Release Tools Bot mentioned in issue gitlab-org/release/tasks#5615 (closed)
mentioned in issue gitlab-org/release/tasks#5615 (closed)
- Steve Abrams mentioned in epic &873 (closed)
mentioned in epic &873 (closed)