Jobs timing out and 100s of test failures on master pipelines
There are a lot test failures in Gitlab master pipelines.
Example pipeline: https://gitlab.com/gitlab-org/gitlab/-/pipelines/1256994352
Some jobs are timing out like https://gitlab.com/gitlab-org/gitlab/-/jobs/6648508961
[Apr 17 2024 13:50:06 UTC (QA Tests)] INFO -- ==> Retrieved a QA::Resource::GroupRunner with name 'runner-for-compliance-pipeline-xm5yael7' via api in 2.03 seconds
[Apr 17 2024 13:50:08 UTC (QA Tests)] INFO -- ==> Built a QA::Resource::Project with full_path 'gitlab-qa-sandbox-group-4/compliance-pipeline-xm5yael7/pipeline-project-86e21bf8efeec61d' via api in 1.99 seconds
[Apr 17 2024 13:50:09 UTC (QA Tests)] INFO -- ==> Built a QA::Resource::Repository::Commit via api in 0.86 seconds
[Apr 17 2024 13:50:10 UTC (QA Tests)] INFO -- ==> Built a QA::Resource::Project with full_path 'gitlab-qa-sandbox-group-4/compliance-pipeline-xm5yael7/compliant-project-f4546d57cd4588a1' via api in 1.22 seconds
[Apr 17 2024 13:50:10 UTC (QA Tests)] INFO -- ==> Built a QA::EE::Resource::ComplianceFramework with name 'Est est pariatur repellat.' via api in 0.11 seconds
[Apr 17 2024 13:50:12 UTC (QA Tests)] INFO -- ==> Built a QA::Resource::Repository::Commit via api in 1.07 seconds
[Apr 17 2024 13:50:13 UTC (QA Tests)] INFO -- ==> Built a QA::Resource::Repository::Commit via api in 0.73 seconds
[Apr 17 2024 13:51:15 UTC (QA Tests)] WARN -- WARN: Screenshot could not be saved. `page.current_path` is empty.
runs that pipeline in a different project that has the compliance framework assigned (FAILED - 14)
Create
Push Rules
using non signed commits
[Apr 17 2024 13:51:16 UTC (QA Tests)] INFO -- filling :username-field with "root"
[Apr 17 2024 13:51:16 UTC (QA Tests)] INFO -- clicking the coordinates of :username-field
ERROR: Job failed: execution took longer than 1h30m0s seconds
Some other jobs are failing, for example https://gitlab.com/gitlab-org/gitlab/-/jobs/6648509028
Failures:
1) Systems Gitaly backend node recovery recovers from dataloss
Failure/Error:
Resource::Repository::ProjectPush.fabricate! do |push|
push.project = project
push.new_branch = false
push.commit_message = 'pushed after failover'
push.file_name = 'new_file'
push.file_content = 'new file'
end
QA::Resource::Events::EventNotFoundError:
Timed out waiting for event
It seems like the tests are unable to push to any repositories. This needs investigation.
Edited by Ievgen Chernikov