Skip to content

Production/Staging/Staging-Ref E2E pipelines - QA user gets blocked and API requests return 403, can block deployments

Around 3:30 PM UTC on 09/22/2022, multiple errors were reported to #qa-staging similar to the following:

An unexpected error occurred while reporting test results in issues.
The error occurred in job: https://ops.gitlab.net/gitlab-org/quality/staging/-/jobs/8065618
Gitlab::Error::Forbidden Server responded with code 403, message: 403 Forbidden - Your account has been blocked.. Request URI: https://gitlab.com/api/v4/projects/gitlab-org%2Fgitlab/issues/347972

It lasted for approximately 10 minutes before resolving on its own

This error message originates from https://gitlab.com/gitlab-org/gitlab-qa/-/blob/df1b220c20f0c49865d09ee84ebbcedbc3bfc8f5/lib/gitlab/qa/report/gitlab_issue_client.rb#L151 and uses the GITLAB_QA_PRODUCTION_ACCESS_TOKEN to report the issues

Around the same time, tests started failing in production as well due to multiple 403s / invalid login and password / blocked account errors for the gitlab-qa user:

These tests later passed on a retry

403 Error Example:

Manage Group with members can be shared with another group with correct access level
     Failure/Error:
       Resource::Group.fabricate_via_api! do |group|
         group.path = "target-group-with-project_#{SecureRandom.hex(8)}"
       end
     
     QA::Resource::Errors::ResourceFabricationFailedError:
       Fabrication of QA::Resource::Sandbox using the API failed (403) with `{"message":"403 Forbidden - Your account has been blocked."}`.
Invalid Login / Password Example 
Manage project insights page behaves like default insights page displays issues and merge requests dashboards
     Failure/Error: Flow::Login.sign_in
/builds/gitlab-org/quality/canary/.gems/bin/gitlab-qa: Docker command `docker run -t --rm --net=bridge --env QA_CAN_TEST_GIT_PROTOCOL_V2="$QA_CAN_TEST_GIT_PROTOCOL_V2" ...` failed! \e[31m✘\e[0m (Gitlab::QA::Docker::Shellout::StatusError)
     
     QA::Page::Validatable::PageValidationError:
       navbar did not appear on QA::Page::Main::Menu as expected
     Shared Example Group: "default insights page" called from ./qa/specs/features/ee/browser_ui/1_manage/insights/default_insights_spec.rb:45
     # ./qa/page/validatable.rb:15:in `block in validate_elements_present!'
     # ./qa/page/validatable.rb:13:in `each'
     # ./qa/page/validatable.rb:13:in `validate_elements_present!'
     # ./qa/page/main/login.rb:210:in `sign_in_using_gitlab_credentials'
     # ./qa/page/main/login.rb:67:in `block in sign_in_using_credentials'
     # ./qa/page/main/login.rb:57:in `sign_in_using_credentials'
     # ./qa/flow/login.rb:34:in `block in sign_in'
     # ./qa/scenario/actable.rb:16:in `perform'
     # ./qa/flow/login.rb:30:in `sign_in'
     # ./qa/specs/features/ee/browser_ui/1_manage/insights/default_insights_spec.rb:33:in `block (3 levels) in <module:QA>'
     # ./qa/specs/spec_helper.rb:149:in `block (2 levels) in <top (required)>'
     # ./qa/specs/runner.rb:65:in `perform'
     # ./qa/scenario/template.rb:10:in `block in perform'
     # ./qa/scenario/template.rb:8:in `tap'
     # ./qa/scenario/template.rb:8:in `perform'
     # ./qa/scenario/template.rb:50:in `perform'
     # ./qa/scenario/template.rb:10:in `block in perform'
     # ./qa/scenario/template.rb:8:in `tap'
     # ./qa/scenario/template.rb:8:in `perform'
     # ./qa/scenario/bootable.rb:51:in `launch!'

manage_project_insights_page_behaves_like_default_insights_page_displays_issues_and_merge_requests_d

Investigation

Possibilities

Infra issue opened to ask for help: https://gitlab.com/gitlab-com/gl-infra/reliability/-/issues/17473

Edited by Mark Lapierre