Production/Staging/Staging-Ref E2E pipelines - QA user gets blocked and API requests return 403, can block deployments
Around 3:30 PM UTC on 09/22/2022, multiple errors were reported to #qa-staging
similar to the following:
An unexpected error occurred while reporting test results in issues.
The error occurred in job: https://ops.gitlab.net/gitlab-org/quality/staging/-/jobs/8065618
Gitlab::Error::Forbidden Server responded with code 403, message: 403 Forbidden - Your account has been blocked.. Request URI: https://gitlab.com/api/v4/projects/gitlab-org%2Fgitlab/issues/347972
It lasted for approximately 10 minutes before resolving on its own
This error message originates from https://gitlab.com/gitlab-org/gitlab-qa/-/blob/df1b220c20f0c49865d09ee84ebbcedbc3bfc8f5/lib/gitlab/qa/report/gitlab_issue_client.rb#L151 and uses the GITLAB_QA_PRODUCTION_ACCESS_TOKEN
to report the issues
Around the same time, tests started failing in production as well due to multiple 403s / invalid login and password / blocked account errors for the gitlab-qa
user:
- gitlab-org/quality/testcase-sessions#56079 (closed)
- gitlab-org/quality/testcase-sessions#56080 (closed)
- gitlab-org/quality/testcase-sessions#56083 (closed)
These tests later passed on a retry
403 Error Example:
Manage Group with members can be shared with another group with correct access level
Failure/Error:
Resource::Group.fabricate_via_api! do |group|
group.path = "target-group-with-project_#{SecureRandom.hex(8)}"
end
QA::Resource::Errors::ResourceFabricationFailedError:
Fabrication of QA::Resource::Sandbox using the API failed (403) with `{"message":"403 Forbidden - Your account has been blocked."}`.
Invalid Login / Password Example
Manage project insights page behaves like default insights page displays issues and merge requests dashboards
Failure/Error: Flow::Login.sign_in
/builds/gitlab-org/quality/canary/.gems/bin/gitlab-qa: Docker command `docker run -t --rm --net=bridge --env QA_CAN_TEST_GIT_PROTOCOL_V2="$QA_CAN_TEST_GIT_PROTOCOL_V2" ...` failed! \e[31m✘\e[0m (Gitlab::QA::Docker::Shellout::StatusError)
QA::Page::Validatable::PageValidationError:
navbar did not appear on QA::Page::Main::Menu as expected
Shared Example Group: "default insights page" called from ./qa/specs/features/ee/browser_ui/1_manage/insights/default_insights_spec.rb:45
# ./qa/page/validatable.rb:15:in `block in validate_elements_present!'
# ./qa/page/validatable.rb:13:in `each'
# ./qa/page/validatable.rb:13:in `validate_elements_present!'
# ./qa/page/main/login.rb:210:in `sign_in_using_gitlab_credentials'
# ./qa/page/main/login.rb:67:in `block in sign_in_using_credentials'
# ./qa/page/main/login.rb:57:in `sign_in_using_credentials'
# ./qa/flow/login.rb:34:in `block in sign_in'
# ./qa/scenario/actable.rb:16:in `perform'
# ./qa/flow/login.rb:30:in `sign_in'
# ./qa/specs/features/ee/browser_ui/1_manage/insights/default_insights_spec.rb:33:in `block (3 levels) in <module:QA>'
# ./qa/specs/spec_helper.rb:149:in `block (2 levels) in <top (required)>'
# ./qa/specs/runner.rb:65:in `perform'
# ./qa/scenario/template.rb:10:in `block in perform'
# ./qa/scenario/template.rb:8:in `tap'
# ./qa/scenario/template.rb:8:in `perform'
# ./qa/scenario/template.rb:50:in `perform'
# ./qa/scenario/template.rb:10:in `block in perform'
# ./qa/scenario/template.rb:8:in `tap'
# ./qa/scenario/template.rb:8:in `perform'
# ./qa/scenario/bootable.rb:51:in `launch!'
Investigation
Possibilities
- Rate-limiting that leads to a temporary block? https://docs.gitlab.com/ee/user/gitlab_com/index.html#gitlabcom-specific-rate-limits
- A temporary IP block: https://docs.gitlab.com/ee/user/gitlab_com/index.html#ip-blocks
Infra issue opened to ask for help: https://gitlab.com/gitlab-com/gl-infra/reliability/-/issues/17473
Edited by Mark Lapierre