QA smoke test fails to create a Project Resource using the API

Summary

During the review-qa-smoke job in a pipeline, four specific cases consistently fail no matter how many times the job is retried.

/qa/specs/features/browser_ui/1_manage/project/create_project_spec.rb fails with this error:

2) Manage Project creation user creates a new project
     Failure/Error: expect(page).to have_content(created_project.name)
       expected to find text "awesome-project-504638cb6d3fa5dc" in "503 Whoops, GitLab is currently unavailable. Try refreshing the page, or going back and attempting the action again. Please contact your GitLab administrator if this problem persists. Go back"
     # ./qa/specs/features/browser_ui/1_manage/project/create_project_spec.rb:15:in `block (3 levels) in <module:QA>'
     # /usr/local/bundle/gems/rspec-retry-0.6.1/lib/rspec/retry.rb:123:in `block in run'
     # /usr/local/bundle/gems/rspec-retry-0.6.1/lib/rspec/retry.rb:110:in `loop'
     # /usr/local/bundle/gems/rspec-retry-0.6.1/lib/rspec/retry.rb:110:in `run'
     # /usr/local/bundle/gems/rspec-retry-0.6.1/lib/rspec_ext/rspec_ext.rb:12:in `run_with_retry'
     # ./spec/spec_helper.rb:46:in `block (2 levels) in <top (required)>'
     # /usr/local/bundle/gems/rspec-retry-0.6.1/lib/rspec/retry.rb:123:in `block in run'
     # /usr/local/bundle/gems/rspec-retry-0.6.1/lib/rspec/retry.rb:110:in `loop'
     # /usr/local/bundle/gems/rspec-retry-0.6.1/lib/rspec/retry.rb:110:in `run'
     # /usr/local/bundle/gems/rspec-retry-0.6.1/lib/rspec_ext/rspec_ext.rb:12:in `run_with_retry'
     # /usr/local/bundle/gems/rspec-retry-0.6.1/lib/rspec/retry.rb:37:in `block (2 levels) in setup'
     # ./qa/specs/runner.rb:56:in `perform'
     # ./qa/scenario/template.rb:10:in `block in perform'
     # ./qa/scenario/template.rb:8:in `tap'
     # ./qa/scenario/template.rb:8:in `perform'
     # ./qa/scenario/template.rb:33:in `perform'
     # ./qa/scenario/template.rb:10:in `block in perform'
     # ./qa/scenario/template.rb:8:in `tap'
     # ./qa/scenario/template.rb:8:in `perform'
     # ./qa/scenario/bootable.rb:28:in `launch!'

And three other cases (qa/specs/features/browser_ui/7_configure/auto_devops/create_project_with_auto_devops_spec.rb, qa/specs/features/browser_ui/4_verify/ci_variable/add_ci_variable_spec.rb, and qa/specs/features/browser_ui/2_plan/issue/create_issue_spec.rb) fail with this error:

3) Verify CI variable support user adds a CI variable
     Failure/Error:
       project = Resource::Project.fabricate! do |project|
         project.name = 'project-with-ci-variables'
         project.description = 'project with CI variables'
       end
     
     QA::Resource::ApiFabricator::ResourceFabricationFailedError:
       Fabrication of QA::Resource::Project using the API failed (400) with `{"message":{"base":["14:channel is in state TRANSIENT_FAILURE"],"limit_reached":[]}}`.
     # ./qa/resource/api_fabricator.rb:71:in `api_post'
     # ./qa/resource/api_fabricator.rb:28:in `fabricate_via_api!'
     # ./qa/resource/project.rb:59:in `rescue in fabricate_via_api!'
     # ./qa/resource/project.rb:56:in `fabricate_via_api!'
     # ./qa/resource/base.rb:46:in `block (2 levels) in fabricate_via_api!'
     # ./qa/resource/base.rb:110:in `log_fabrication'
     # ./qa/resource/base.rb:46:in `block in fabricate_via_api!'
     # ./qa/resource/base.rb:93:in `do_fabricate!'
     # ./qa/resource/base.rb:45:in `fabricate_via_api!'
     # ./qa/resource/base.rb:19:in `fabricate!'
     # ./qa/specs/features/browser_ui/4_verify/ci_variable/add_ci_variable_spec.rb:10:in `block (3 levels) in <module:QA>'
     # /usr/local/bundle/gems/rspec-retry-0.6.1/lib/rspec/retry.rb:123:in `block in run'
     # /usr/local/bundle/gems/rspec-retry-0.6.1/lib/rspec/retry.rb:110:in `loop'
     # /usr/local/bundle/gems/rspec-retry-0.6.1/lib/rspec/retry.rb:110:in `run'
     # /usr/local/bundle/gems/rspec-retry-0.6.1/lib/rspec_ext/rspec_ext.rb:12:in `run_with_retry'
     # ./spec/spec_helper.rb:46:in `block (2 levels) in <top (required)>'
     # /usr/local/bundle/gems/rspec-retry-0.6.1/lib/rspec/retry.rb:123:in `block in run'
     # /usr/local/bundle/gems/rspec-retry-0.6.1/lib/rspec/retry.rb:110:in `loop'
     # /usr/local/bundle/gems/rspec-retry-0.6.1/lib/rspec/retry.rb:110:in `run'
     # /usr/local/bundle/gems/rspec-retry-0.6.1/lib/rspec_ext/rspec_ext.rb:12:in `run_with_retry'
     # /usr/local/bundle/gems/rspec-retry-0.6.1/lib/rspec/retry.rb:37:in `block (2 levels) in setup'
     # ./qa/specs/runner.rb:56:in `perform'
     # ./qa/scenario/template.rb:10:in `block in perform'
     # ./qa/scenario/template.rb:8:in `tap'
     # ./qa/scenario/template.rb:8:in `perform'
     # ./qa/scenario/template.rb:33:in `perform'
     # ./qa/scenario/template.rb:10:in `block in perform'
     # ./qa/scenario/template.rb:8:in `tap'
     # ./qa/scenario/template.rb:8:in `perform'
     # ./qa/scenario/bootable.rb:28:in `launch!'
     # ------------------
     # --- Caused by: ---
     # QA::Resource::ApiFabricator::ResourceNotFoundError:
     #   Resource at https://gitlab-review-9688-fe-mr-qjvyas.gitlab-review.app/api/v4/projects/gitlab-qa-sandbox-group%2Fqa-test-2019-06-04-22-14-15-96364903e7475df1%2Fproject-with-ci-variables-7f26ae4651f60e33?private_token=kxmdXdxxSW-rHZ1Ayhzv could not be found (404): `{"message":"404 Project Not Found"}`.
     #   ./qa/resource/api_fabricator.rb:59:in `api_get_from'

Steps to reproduce

Unknown

What is the current bug behavior?

review-qa-smoke fails due to errors seemingly unrelated to the merge request's changes

What is the expected correct behavior?

review-qa-smoke should pass, pass when retried, or give actionable error messages

Relevant logs and/or screenshots

Here are two affected jobs from separate pipelines for merge requests:

Output of checks

This bug happens on GitLab.com

Results of GitLab environment info

Expand for output related to GitLab environment info 
(For installations with omnibus-gitlab package run and paste the output of:
sudo gitlab-rake gitlab:env:info)


(For installations from source run and paste the output of:
sudo -u git -H bundle exec rake gitlab:env:info RAILS_ENV=production)

Results of GitLab application Check

Expand for output related to the GitLab application check 
(For installations with omnibus-gitlab package run and paste the output of:
sudo gitlab-rake gitlab:check SANITIZE=true)


(For installations from source run and paste the output of:
sudo -u git -H bundle exec rake gitlab:check RAILS_ENV=production SANITIZE=true)


(we will only investigate if the tests are passing)

Possible fixes

It seems like project creation is failing at create_project_spec.rb#L10, then the other three are failing because of that.

Edited by Miranda Fluharty