Backport Testing - Issues
LATEST PIPELINE: https://gitlab.com/gitlab-org/gitlab-qa/-/pipelines/400612434
An attempt was made to run a package-and-qa
pipeline against a 13.12.13 backport
- Pipeline: https://gitlab.com/gitlab-org/gitlab-qa/-/pipelines/399736914 triggered with these values:
GitLab QA branch: v7.3.1
RELEASE registry.gitlab.com/gitlab-org/build/omnibus-gitlab-mirror/gitlab-ee:b09b8ff9a51ccededa26115a6d08dcf385fbf04f
QA_IMAGE registry.gitlab.com/gitlab-org/gitlab/gitlab-ee-qa:13-12-13-latest-commit
Not known issues
-
🚧 - 500 error (Point 3 underFull details
below) ingroup_audit_logs_2_spec.rb
spec✅ -
Tested that audit logs functionality works manually
-
looks like caused by actual bug in %13.12 -
🐛 #333389 (closed) (see more details below), but the fix for #333389 (closed) (omnibus-gitlab!5310 (merged)) was merged into 13.12.3.
-
-
ee:image
(Point 7 underFull details
below): https://gitlab.com/gitlab-org/gitlab-qa/-/jobs/1741134720 -✅ can be ignored due to below details:- GitLab QA docs say that
Test::Omnibus::Image
checks that GitLab Docker container can start without any error. As we can see in other jobs where GitLab container is started, the image is working. The difference between starting container inee:image
andee:instance
jobs appears to be related to--net bridge
config for docker run command and it seems that it causes to refuse connection to check GitLab instance, but overall as we can see in the job logs - GitLab is reconfigured correctly and as we can see in other jobs asee:instance
GitLab instance is working.
- GitLab QA docs say that
Full details:
-
gitaly-cluster
https://gitlab.com/gitlab-org/gitlab-qa-mirror/-/jobs/1736050124 fails on- rspec ./qa/specs/features/api/3_create/gitaly/distributed_reads_spec.rb:56 # Create Gitaly Distributed reads when a node is unhealthy does not read from the unhealthy node -
✅ expected to fail due to #322814 (closed) which was fixed in %14.2 based on #332234 (comment 602339067) - rspec ./qa/specs/features/api/3_create/gitaly/praefect_replication_queue_spec.rb:22 # Create Gitaly Cluster replication queue allows replication of different repository after interruption -
✅ Same failures in https://gitlab.com/gitlab-org/gitlab-qa-mirror/-/jobs/1027459021, reported in #321440 (closed)
1) Create Gitaly Distributed reads when a node is unhealthy does not read from the unhealthy node Got 0 failures and 2 other errors: 1.1) Failure/Error: praefect_manager.wait_for_replication(project.id) QA::Support::Repeater::WaitExceededError: Wait condition not met after 60 seconds ... 1.2) Failure/Error: praefect_manager.wait_for_reliable_connection QA::Support::Repeater::WaitExceededError: Wait condition not met after 120 seconds ... 2) Create Gitaly Cluster replication queue allows replication of different repository after interruption Got 0 failures and 2 other errors: 2.1) Failure/Error: Resource::Project.fabricate! do |project| project.name = "gitaly_cluster" project.initialize_with_readme = true end QA::Resource::ApiFabricator::ResourceFabricationFailedError: Fabrication of QA::Resource::Project using the API failed (400) with `{"message":{"base":["14:DNS resolution failed. debug_error_string:{\"created\":\"@1635784983.550597941\",\"description\":\"Failed to pick subchannel\",\"file\":\"src/core/ext/filters/client_channel/client_channel.cc\",\"file_line\":3952,\"referenced_errors\":[{\"created\":\"@1635784983.550586025\",\"description\":\"Resolver transient failure\",\"file\":\"src/core/ext/filters/client_channel/resolving_lb_policy.cc\",\"file_line\":216,\"referenced_errors\":[{\"created\":\"@1635784983.550583013\",\"description\":\"DNS resolution failed\",\"file\":\"src/core/ext/filters/client_channel/resolver/dns/c_ares/dns_resolver_ares.cc\",\"file_line\":375,\"grpc_status\":14,\"referenced_errors\":[{\"created\":\"@1635784983.550531488\",\"description\":\"C-ares status is not ARES_SUCCESS qtype=A name=praefect.test is_balancer=0: Domain name not found\",\"file\":\"src/core/ext/filters/client_channel/resolver/dns/c_ares/grpc_ares_wrapper.cc\",\"file_line\":287}]}]}]}"],"limit_reached":[]}}`. ... 2.2) Failure/Error: praefect_manager.wait_for_reliable_connection QA::Support::Repeater::WaitExceededError: Wait condition not met after 60 seconds
- rspec ./qa/specs/features/api/3_create/gitaly/distributed_reads_spec.rb:56 # Create Gitaly Distributed reads when a node is unhealthy does not read from the unhealthy node -
-
Passing in latest pipeline: https://gitlab.com/gitlab-org/gitlab-qa/-/pipelines/400612434.group_saml
https://gitlab.com/gitlab-org/gitlab-qa-mirror/-/jobs/1736146231 failed on✅ expected to fail in 13.12 - QA failure #330957 (closed) was fixed in %14.0Click to expand details
Failed examples: rspec ./qa/specs/features/ee/browser_ui/1_manage/group/group_saml_enforced_sso_new_account_spec.rb:35 # Manage Group SAML SSO - Enforced SSO creates a new account automatically and allows to leave group and join again
1) Manage Group SAML SSO - Enforced SSO creates a new account automatically and allows to leave group and join again Failure/Error: QA::Flow::User.confirm_user(user.email) Capybara::ElementNotFound: Unable to find css "[data-qa-selector=\"confirm_user_button\"],.qa-confirm-user-button"
-
ee:instance
ee:praefect
ee:relative_url
🚧 500 errors - looks like caused by actual bug in %13.12 -🐛 #333389 (closed)- rspec ./qa/specs/features/api/1_manage/project_access_token_spec.rb:25 # Manage Project access token for the same project can be used to commit via the API
- rspec './qa/specs/features/ee/browser_ui/1_manage/group/group_audit_logs_2_spec.rb[1:1:4:1:1]
fails due to 500 error on access token generation, looking into logs
3) Manage Project access token for the same project can be used to create a file via the project API Failure/Error: @project_access_token = QA::Resource::ProjectAccessToken.fabricate_via_api! QA::Resource::ApiFabricator::InternalServerError: Failed to GET http://gitlab-ee-d64f6013.test/api/v4/groups/gitlab-qa-sandbox-group%2Fqa-test-2021-11-01-18-34-39-9ddb376f74aa9e02?private_token=[****] - (500): `{"message":"500 Internal Server Error"}`. # ./qa/resource/api_fabricator.rb:90:in `api_get_from'
- rspec ./qa/specs/features/ee/api/2_plan/epics_milestone_dates_spec.rb:19 # Plan Epics milestone dates API changes epic dates when updating milestones
1) Plan Epics milestone dates API changes epic dates when updating milestones Failure/Error: @group_id = Resource::Group.fabricate_via_api!.id QA::Resource::ApiFabricator::InternalServerError: Failed to GET http://gitlab-ee-ffab82ba.test/api/v4/groups/gitlab-qa-sandbox-group%2Fqa-test-2021-11-01-18-34-46-714c7f1fd2553027?private_token=[****] - (500): `{"message":"500 Internal Server Error"}`. # ./qa/resource/api_fabricator.rb:90:in `api_get_from'
500 error from logs:
"severity": "ERROR", "time": "2021-11-01T18:29:08.335Z", "correlation_id": "b9e3b13e6fb44c7d926455d2b457edd4", "exception.class": "Sidekiq::Worker::EnqueueFromTransactionError", "exception.message": "`AuthorizedProjectUpdate::ProjectCreateWorker.perform_async` cannot be called inside a transaction as this can lead to\nrace conditions when the worker runs before the transaction is committed and\ntries to access a model that has not been saved yet.\n\nUse an `after_commit` hook, or include `AfterCommitQueue` and use a `run_after_commit` block instead.\n", "exception.backtrace": [ "config/initializers/forbid_sidekiq_in_transactions.rb:25:in `block (2 levels) in <module:NoEnqueueingFromTransactions>'", "app/services/projects/create_service.rb:132:in `setup_authorizations'", "app/services/projects/create_service.rb:104:in `after_create_actions'", "ee/app/services/ee/projects/create_service.rb:48:in `after_create_actions'
-
. !62000 (merged) was not released in 13.12, which is why this was failing. See #344577 (comment 720946892). Once we merged !62000 (merged) into 13-12-13-with-fixes, and ran a new pipeline, this failure disappeared.ee:instance
https://gitlab.com/gitlab-org/gitlab-qa-mirror/-/jobs/1736191815 /ee:praefect 2/10
https://gitlab.com/gitlab-org/gitlab-qa-mirror/-/jobs/1736050108 push rules -❓ Same failures in https://ops.gitlab.net/gitlab-org/quality/staging/-/jobs/3887209 and https://gitlab.com/gitlab-org/gitlab-qa-mirror/-/jobs/1268464617, reported in #298755 (closed), #299081 (closed), but should've been fixed with !62000 (merged)Click to expand details
rspec ./qa/specs/features/ee/browser_ui/3_create/repository/push_rules_spec.rb:82 # Create Push Rules using non signed commits restricts committing files with secrets rspec ./qa/specs/features/ee/browser_ui/3_create/repository/push_rules_spec.rb:39 # Create Push Rules using non signed commits restricts files by name and size
2) Create Push Rules using non signed commits restricts committing files with secrets Failure/Error: expect do push commit_message: commit_message, branch: branch, file: file, user: user, tag: tag, gpg: gpg end.to raise_error(QA::Support::Run::CommandError, /#{error}/) expected QA::Support::Run::CommandError with message matching /File\ name\ id_rsa\ was\ prohibited\ by\ the\ pattern\ id_rsa\$/, got #<QA::Support::Run::CommandError: The command HOME="/tmp/qa-netrc-credentials/23" git push http://roo...tlab-qa-sandbox-group/qa-test-2021-11-01-16-21-01-729c429bdfe6afb7/push_rules-541c092a4058e97c.git'> with backtrace: ... 3) Create Push Rules using non signed commits restricts files by name and size Failure/Error: expect do push commit_message: commit_message, branch: branch, file: file, user: user, tag: tag, gpg: gpg end.to raise_error(QA::Support::Run::CommandError, /#{error}/) expected QA::Support::Run::CommandError with message matching /File\ name\ denied_file\ was\ prohibited\ by\ the\ pattern\ denied_file/, got #<QA::Support::Run::CommandError: The command HOME="/tmp/qa-netrc-credentials/23" git push http://roo...tlab-qa-sandbox-group/qa-test-2021-11-01-16-21-01-729c429bdfe6afb7/push_rules-541c092a4058e97c.git'> with backtrace:
-
Has not appeared in latest pipeline: https://gitlab.com/gitlab-org/gitlab-qa/-/pipelines/400612434.object_storage_registry_tls
https://gitlab.com/gitlab-org/gitlab-qa-mirror/-/jobs/1736050079 /registry
https://gitlab.com/gitlab-org/gitlab-qa-mirror/-/jobs/1736050085 failure -✅ expected to fail due to an open QA failure #344562 (closed)Click to expand
NameError: uninitialized constant QA::Scenario::Test::Integration::RegistryTLS
NameError: uninitialized constant QA::Scenario::Test::Integration::Registry
Looks same issue as #344562 (closed)
-
ee:update
https://gitlab.com/gitlab-org/gitlab-qa-mirror/-/jobs/1736049993 chef script failure✅ expected to fail as it trying to rollback from 14.4 to 13.12 (internal DJ's comment)Running handlers: There was an error running gitlab-ctl reconfigure: rails_migration[gitlab-rails] (gitlab::database_migrations line 54) had an error: Mixlib::ShellOut::ShellCommandFailed: bash[migrate gitlab-rails database] (/opt/gitlab/embedded/cookbooks/cache/cookbooks/gitlab/resources/rails_migration.rb line 16) had an error: Mixlib::ShellOut::ShellCommandFailed: Expected process to exit with [0], but received '1' ---- Begin output of "bash" "/tmp/chef-script20211101-38-w1tn37" ---- STDOUT: rake aborted! StandardError: An error has occurred, all later migrations canceled: PG::UndefinedColumn: ERROR: column "artifacts_file" does not exist LINE 1: .../ SELECT "ci_builds"."id" FROM "ci_builds" WHERE (artifacts_...
From Slack discussion, it seems this would be expected to fail https://gitlab.slack.com/archives/C3JJET4Q6/p1635787631229800?thread_ts=1635767069.205800&cid=C3JJET4Q6
@twk3's Slack message reproduced here:
The current gitlab-qa project defaults to main as the default branch for the qa runtime env, which won't work against GitLab 13.12. And also the update tests end up being a rollback from latest rather than an update. And I don't think you can rollback from 14.4 to 13.12, which likely explains the migrations failures in those tests.
-
ee:image
: https://gitlab.com/gitlab-org/gitlab-qa/-/jobs/1741134720GitLab instance failed to start
❓ Waiting for GitLab at `http://docker:49154/help` to become available Failed to open TCP connection to docker:49154 (Connection refused - connect(2) for "docker" port 49154).