QA tests failing when run against a Review App
QA tests keep failing when run against a Review App.
I discovered this while trying to resolve failures that occur when tests are run in parallel, and found that some also fail when the tests are not run in parallel.
For example: https://gitlab.com/gitlab-org/gitlab-ce/-/jobs/259700349
Finished in 50 minutes 46 seconds (files took 2.47 seconds to load)
60 examples, 19 failures, 9 pending
Failed examples:
rspec ./qa/specs/features/browser_ui/3_create/wiki/create_edit_clone_push_wiki_spec.rb:6 # Create Wiki management user creates, edits, clones, and pushes to the wiki
rspec ./qa/specs/features/browser_ui/3_create/repository/add_list_delete_branches_spec.rb:59 # Create Create, list, and delete branches via web lists branches correctly after CRUD operations
rspec './qa/specs/features/browser_ui/3_create/web_ide/add_file_template_spec.rb[1:1:1]' # Create Web IDE file templates user adds .gitignore via file template Android
rspec './qa/specs/features/browser_ui/3_create/web_ide/add_file_template_spec.rb[1:1:2]' # Create Web IDE file templates user adds .gitlab-ci.yml via file template Julia
rspec './qa/specs/features/browser_ui/3_create/web_ide/add_file_template_spec.rb[1:1:3]' # Create Web IDE file templates user adds Dockerfile via file template Python
rspec './qa/specs/features/browser_ui/3_create/web_ide/add_file_template_spec.rb[1:1:4]' # Create Web IDE file templates user adds LICENSE via file template Mozilla Public License 2.0
rspec ./qa/specs/features/browser_ui/3_create/repository/create_edit_delete_file_via_web_spec.rb:6 # Create Files management user creates, edits and deletes a file via the Web
rspec ./qa/specs/features/browser_ui/3_create/merge_request/merge_merge_request_from_fork_spec.rb:6 # Create Merge request creation from fork user forks a project, submits a merge request and maintainer merges it
rspec ./qa/specs/features/browser_ui/3_create/repository/push_over_http_spec.rb:6 # Create Git push over HTTP user pushes code to the repository
rspec ./qa/specs/features/browser_ui/3_create/merge_request/squash_merge_request_spec.rb:6 # Create Merge request squashing user squashes commits while merging
rspec ./qa/specs/features/browser_ui/1_manage/group/transfer_project_spec.rb:6 # Manage Project transfer between groups user transfers a project between groups
rspec ./qa/specs/features/api/3_create/repository/files_spec.rb:14 # API basics user creates a project with a file and deletes them afterwards
rspec ./qa/specs/features/api/3_create/repository/project_archive_compare_spec.rb:28 # Create Compare archives of different user projects with the same name and check they're different download archives of each user project then check they are different
rspec './qa/specs/features/browser_ui/3_create/repository/add_file_template_spec.rb[1:1:1]' # Create File templates user adds .gitignore via file template Android
rspec './qa/specs/features/browser_ui/3_create/repository/add_file_template_spec.rb[1:1:2]' # Create File templates user adds .gitlab-ci.yml via file template Julia
rspec './qa/specs/features/browser_ui/3_create/repository/add_file_template_spec.rb[1:1:3]' # Create File templates user adds Dockerfile via file template Python
rspec './qa/specs/features/browser_ui/3_create/repository/add_file_template_spec.rb[1:1:4]' # Create File templates user adds LICENSE via file template Mozilla Public License 2.0
rspec ./qa/specs/features/browser_ui/3_create/merge_request/view_merge_request_diff_patch_spec.rb:16 # Create Download merge request patch and diff user views merge request email patches
rspec ./qa/specs/features/browser_ui/3_create/merge_request/view_merge_request_diff_patch_spec.rb:25 # Create Download merge request patch and diff user views merge request plain diff
/usr/local/bundle/gems/gitlab-qa-4.1.0/lib/gitlab/qa/docker/shellout.rb:31:in `block in execute!': Docker command `docker run -t --rm --net=bridge --env GITLAB_ADMIN_USERNAME="$GITLAB_ADMI
That job is like the old review-qa-all
job, i.e., it runs all the tests in series.
But even individual jobs from parallelized review-qa-all
fail when run in isolation. E.g.: https://gitlab.com/gitlab-org/gitlab-ce/-/jobs/259730962
Finished in 10 minutes 14 seconds (files took 2.37 seconds to load)
6 examples, 2 failures, 1 pending
Failed examples:
rspec ./qa/specs/features/browser_ui/3_create/repository/add_list_delete_branches_spec.rb:59 # Create Create, list, and delete branches via web lists branches correctly after CRUD operations
rspec ./qa/specs/features/api/3_create/repository/files_spec.rb:14 # API basics user creates a project with a file and deletes them afterwards
When I ran that job it was the only job running.
The tests pass when run in package-and-qa
.