Several job timeouts occurring while some Create specs are running
Summary
There have been an increased occurrence of jobs timing out specifically when certain Create specs are running
Create File Locking locks a file and tries to push as a second user
Create Push Rules using non signed commits restricts files by name and size
Create Push Rules using non signed commits allows an unrestricted push
Create Push Rules using non signed commits restricts committing files with secrets
Create File Locking creates a merge request and fails to merge
Failing job(s):
Create File Locking locks a file and tries to push as a second user
- https://ops.gitlab.net/gitlab-org/quality/canary/-/jobs/13521648
- https://ops.gitlab.net/gitlab-org/quality/production/-/jobs/13520466
- https://ops.gitlab.net/gitlab-org/quality/canary/-/jobs/13519297
- https://ops.gitlab.net/gitlab-org/quality/staging-canary/-/jobs/13522157
- https://ops.gitlab.net/gitlab-org/quality/staging-canary/-/jobs/13516470
Create Push Rules using non signed commits restricts files by name and size
- https://ops.gitlab.net/gitlab-org/quality/canary/-/jobs/13520313
- https://ops.gitlab.net/gitlab-org/quality/canary/-/jobs/13519213
- https://ops.gitlab.net/gitlab-org/quality/canary/-/jobs/13514356
- https://ops.gitlab.net/gitlab-org/quality/canary/-/jobs/13510155
- https://ops.gitlab.net/gitlab-org/quality/canary/-/jobs/13508680
- https://ops.gitlab.net/gitlab-org/quality/staging-canary/-/jobs/13523536
- https://ops.gitlab.net/gitlab-org/quality/staging-canary/-/jobs/13519808
Create Push Rules using non signed commits allows an unrestricted push
- https://ops.gitlab.net/gitlab-org/quality/staging-canary/-/pipelines/3120617
- https://ops.gitlab.net/gitlab-org/quality/staging-canary/-/jobs/13541989
Create Push Rules using non signed commits restricts committing files with secrets
Create File Locking creates a merge request and fails to merge
- https://ops.gitlab.net/gitlab-org/quality/staging/-/jobs/13523934
- https://ops.gitlab.net/gitlab-org/quality/staging-canary/-/jobs/13522593
Failing spec(s):
- https://gitlab.com/gitlab-org/gitlab/-/blob/master/qa/qa/specs/features/ee/browser_ui/3_create/repository/file_locking_spec.rb#L35
- https://gitlab.com/gitlab-org/gitlab/-/blob/master/qa/qa/specs/features/ee/browser_ui/3_create/repository/push_rules_spec.rb#L36
- https://gitlab.com/gitlab-org/gitlab/-/blob/master/qa/qa/specs/features/ee/browser_ui/3_create/repository/file_locking_spec.rb#L69
Corresponding test case(s):
- https://gitlab.com/gitlab-org/gitlab/-/quality/test_cases/347784
- https://gitlab.com/gitlab-org/gitlab/-/quality/test_cases/347768
- https://gitlab.com/gitlab-org/gitlab/-/quality/test_cases/347770
Stack trace
Create File Locking locks a file and tries to push as a second user
With this spec, in the instances it failed, 2 of them failed on the branch creation and the other 2 failed a bit later when it tried to lock the file then finally the last instance was on project creation.
- https://ops.gitlab.net/gitlab-org/quality/canary/-/jobs/13521648
- https://ops.gitlab.net/gitlab-org/quality/production/-/jobs/13520466
[Apr 12 2024 19:09:28 UTC (QA Tests)] INFO -- Starting test: Create File Locking locks a directory and tries to push as a second user
[Apr 12 2024 19:09:29 UTC (QA Tests)] INFO -- filling :username-field with "gitlab-qa"
[Apr 12 2024 19:09:29 UTC (QA Tests)] INFO -- clicking the coordinates of :username-field
[Apr 12 2024 19:09:30 UTC (QA Tests)] INFO -- filling :password-field with "*****"
[Apr 12 2024 19:09:30 UTC (QA Tests)] INFO -- clicking the coordinates of :password-field
[Apr 12 2024 19:09:30 UTC (QA Tests)] INFO -- clicking :sign-in-button
[Apr 12 2024 19:09:32 UTC (QA Tests)] WARN -- Potentially Slow Code 'click_element sign-in-button' took 2.403s
[Apr 12 2024 19:09:36 UTC (QA Tests)] INFO -- ==> Retrieved a QA::Resource::Sandbox with full_path 'gitlab-qa-sandbox-group-6' via api in 0.7 seconds
[Apr 12 2024 19:09:36 UTC (QA Tests)] INFO -- ==> Retrieved a QA::Resource::Group with full_path 'gitlab-qa-sandbox-group-6/qa-test-2024-04-12-19-06-26-1083d43f4d3b79f7' via api in 1.13 seconds
[Apr 12 2024 19:09:38 UTC (QA Tests)] INFO -- ==> Built a QA::Resource::Project with full_path 'gitlab-qa-sandbox-group-6/qa-test-2024-04-12-19-06-26-1083d43f4d3b79f7/file_locking-12fd83e68373b84b' via api in 3.41 seconds
[Apr 12 2024 19:09:40 UTC (QA Tests)] INFO -- QA::Resource::Project - wait_for_push with commit message "This is a test commit"
[Apr 12 2024 19:09:41 UTC (QA Tests)] INFO -- ==> Built a QA::Resource::Repository::ProjectPush via browser_ui in 2.7 seconds
[Apr 12 2024 19:09:41 UTC (QA Tests)] INFO -- ==> Retrieved a QA::Resource::User with username 'gitlab-qa-user1' via api in 0.31 seconds
[Apr 12 2024 19:09:42 UTC (QA Tests)] INFO -- ==> Built a QA::Resource::ProjectMember via api in 0.62 seconds
[Apr 12 2024 19:09:42 UTC (QA Tests)] INFO -- ==> Retrieved a QA::Resource::User with username 'gitlab-qa-user2' via api in 0.31 seconds
[Apr 12 2024 19:09:43 UTC (QA Tests)] INFO -- ==> Built a QA::Resource::ProjectMember via api in 1.21 seconds
[Apr 12 2024 19:09:44 UTC (QA Tests)] INFO -- Removed a QA::Resource::ProtectedBranch
[Apr 12 2024 19:09:44 UTC (QA Tests)] WARN -- Resource fabrication http method has not been set properly, assuming :get value!
[Apr 12 2024 19:09:44 UTC (QA Tests)] WARN -- Resource fabrication http method has not been set properly, assuming :get value!
[Apr 12 2024 19:09:44 UTC (QA Tests)] INFO -- ==> Built a QA::Resource::ProtectedBranch via api in 0.84 seconds
Terminated
- https://ops.gitlab.net/gitlab-org/quality/canary/-/jobs/13519297
- https://ops.gitlab.net/gitlab-org/quality/staging-canary/-/jobs/13522157
[Apr 12 2024 15:56:54 UTC (QA Tests)] INFO -- Starting test: Create File Locking locks a file and tries to push as a second user
[Apr 12 2024 15:56:56 UTC (QA Tests)] INFO -- filling :username-field with "gitlab-qa"
[Apr 12 2024 15:56:56 UTC (QA Tests)] INFO -- clicking the coordinates of :username-field
[Apr 12 2024 15:56:56 UTC (QA Tests)] INFO -- filling :password-field with "*****"
[Apr 12 2024 15:56:56 UTC (QA Tests)] INFO -- clicking the coordinates of :password-field
[Apr 12 2024 15:56:57 UTC (QA Tests)] INFO -- clicking :sign-in-button
[Apr 12 2024 15:56:59 UTC (QA Tests)] WARN -- Potentially Slow Code 'click_element sign-in-button' took 2.692s
[Apr 12 2024 15:57:02 UTC (QA Tests)] INFO -- ==> Retrieved a QA::Resource::Sandbox with full_path 'gitlab-qa-sandbox-group-6' via api in 0.23 seconds
[Apr 12 2024 15:57:03 UTC (QA Tests)] INFO -- ==> Retrieved a QA::Resource::Group with full_path 'gitlab-qa-sandbox-group-6/qa-test-2024-04-12-15-56-21-f9a4b380592fe3ef' via api in 0.85 seconds
[Apr 12 2024 15:57:06 UTC (QA Tests)] INFO -- ==> Built a QA::Resource::Project with full_path 'gitlab-qa-sandbox-group-6/qa-test-2024-04-12-15-56-21-f9a4b380592fe3ef/file_locking-1aeff70484e096a5' via api in 4.6 seconds
[Apr 12 2024 15:57:10 UTC (QA Tests)] INFO -- QA::Resource::Project - wait_for_push with commit message "This is a test commit"
[Apr 12 2024 15:57:10 UTC (QA Tests)] INFO -- ==> Built a QA::Resource::Repository::ProjectPush via browser_ui in 3.96 seconds
[Apr 12 2024 15:57:11 UTC (QA Tests)] INFO -- ==> Retrieved a QA::Resource::User with username 'gitlab-qa-user1' via api in 0.29 seconds
[Apr 12 2024 15:57:11 UTC (QA Tests)] INFO -- ==> Built a QA::Resource::ProjectMember via api in 0.68 seconds
[Apr 12 2024 15:57:12 UTC (QA Tests)] INFO -- ==> Retrieved a QA::Resource::User with username 'gitlab-qa-user2' via api in 0.29 seconds
[Apr 12 2024 15:57:12 UTC (QA Tests)] INFO -- ==> Built a QA::Resource::ProjectMember via api in 0.55 seconds
[Apr 12 2024 15:57:13 UTC (QA Tests)] INFO -- Removed a QA::Resource::ProtectedBranch
[Apr 12 2024 15:57:13 UTC (QA Tests)] WARN -- Resource fabrication http method has not been set properly, assuming :get value!
[Apr 12 2024 15:57:13 UTC (QA Tests)] WARN -- Resource fabrication http method has not been set properly, assuming :get value!
[Apr 12 2024 15:57:13 UTC (QA Tests)] INFO -- ==> Built a QA::Resource::ProtectedBranch via api in 1.03 seconds
[Apr 12 2024 15:57:14 UTC (QA Tests)] WARN -- Potentially Slow Code 'has_element? global-search-input' took 1.018s
[Apr 12 2024 15:57:16 UTC (QA Tests)] WARN -- Potentially Slow Code 'has_element? user-profile-link' took 1.034s
[Apr 12 2024 15:57:16 UTC (QA Tests)] INFO -- clicking :user-avatar-content
[Apr 12 2024 15:57:16 UTC (QA Tests)] INFO -- clicking :sign-out-link
[Apr 12 2024 15:57:17 UTC (QA Tests)] WARN -- Potentially Slow Code 'click_element sign-out-link' took 1.282s
[Apr 12 2024 15:57:18 UTC (QA Tests)] INFO -- filling :username-field with "gitlab-qa-user1"
[Apr 12 2024 15:57:18 UTC (QA Tests)] INFO -- clicking the coordinates of :username-field
[Apr 12 2024 15:57:19 UTC (QA Tests)] INFO -- filling :password-field with "*****"
[Apr 12 2024 15:57:19 UTC (QA Tests)] INFO -- clicking the coordinates of :password-field
[Apr 12 2024 15:57:19 UTC (QA Tests)] INFO -- clicking :sign-in-button
[Apr 12 2024 15:57:21 UTC (QA Tests)] WARN -- Potentially Slow Code 'click_element sign-in-button' took 2.033s
[Apr 12 2024 15:57:23 UTC (QA Tests)] INFO -- Visiting QA::Resource::Project at https://gitlab.com/gitlab-qa-sandbox-group-6/qa-test-2024-04-12-15-56-21-f9a4b380592fe3ef/file_locking-1aeff70484e096a5
[Apr 12 2024 15:57:26 UTC (QA Tests)] INFO -- clicking :file-name-link
[Apr 12 2024 15:57:26 UTC (QA Tests)] INFO -- clicking :lock-button
[Apr 12 2024 15:57:27 UTC (QA Tests)] WARN -- Potentially Slow Code 'wait_for_requests ' took 0.551s
[Apr 12 2024 15:57:27 UTC (QA Tests)] WARN -- Potentially Slow Code 'click_element lock-button' took 0.716s
[Apr 12 2024 15:57:27 UTC (QA Tests)] INFO -- clicking :confirm-ok-button
Terminated
[Apr 12 2024 14:08:29 UTC (QA Tests)] INFO -- Starting test: Create File Locking locks a file and tries to push as a second user
[Apr 12 2024 14:08:30 UTC (QA Tests)] INFO -- filling :username-field with "gitlab-qa"
[Apr 12 2024 14:08:30 UTC (QA Tests)] INFO -- clicking the coordinates of :username-field
[Apr 12 2024 14:08:31 UTC (QA Tests)] INFO -- filling :password-field with "*****"
[Apr 12 2024 14:08:31 UTC (QA Tests)] INFO -- clicking the coordinates of :password-field
[Apr 12 2024 14:08:31 UTC (QA Tests)] INFO -- clicking :sign-in-button
[Apr 12 2024 14:08:34 UTC (QA Tests)] WARN -- Potentially Slow Code 'click_element sign-in-button' took 2.811s
[Apr 12 2024 14:08:37 UTC (QA Tests)] INFO -- ==> Retrieved a QA::Resource::Sandbox with full_path 'gitlab-qa-sandbox-group-6' via api in 0.22 seconds
[Apr 12 2024 14:08:37 UTC (QA Tests)] INFO -- ==> Retrieved a QA::Resource::Group with full_path 'gitlab-qa-sandbox-group-6/qa-test-2024-04-12-14-06-10-4fb609697d6f26c2' via api in 0.65 seconds
[Apr 12 2024 14:08:39 UTC (QA Tests)] INFO -- ==> Built a QA::Resource::Project with full_path 'gitlab-qa-sandbox-group-6/qa-test-2024-04-12-14-06-10-4fb609697d6f26c2/file_locking-1112fd2b11fb6c3d' via api in 3.09 seconds
Terminated
Create Push Rules using non signed commits restricts files by name and size
With this spec, all of the times it timed out where right when the test started, potentially the set up before the test runs gets stuck sometimes.
[Apr 12 2024 17:35:19 UTC (QA Tests)] INFO -- Starting test: Create Push Rules using non signed commits restricts files by name and size
Terminated
Create File Locking creates a merge request and fails to merge
With this one, it's also failing at 2 different points in the test. Both of them seem to be failing on resource creation of either a Project or a Branch.
[Apr 12 2024 23:04:22 UTC (QA Tests)] INFO -- Starting test: Create File Locking creates a merge request and fails to merge
[Apr 12 2024 23:04:24 UTC (QA Tests)] INFO -- filling :username-field with "gitlab-qa"
[Apr 12 2024 23:04:24 UTC (QA Tests)] INFO -- clicking the coordinates of :username-field
[Apr 12 2024 23:04:24 UTC (QA Tests)] INFO -- filling :password-field with "*****"
[Apr 12 2024 23:04:24 UTC (QA Tests)] INFO -- clicking the coordinates of :password-field
[Apr 12 2024 23:04:24 UTC (QA Tests)] INFO -- clicking :sign-in-button
[Apr 12 2024 23:04:27 UTC (QA Tests)] WARN -- Potentially Slow Code 'click_element sign-in-button' took 2.58s
[Apr 12 2024 23:04:29 UTC (QA Tests)] INFO -- ==> Retrieved a QA::Resource::Sandbox with full_path 'gitlab-qa-sandbox-group-6' via api in 0.18 seconds
[Apr 12 2024 23:04:30 UTC (QA Tests)] INFO -- ==> Retrieved a QA::Resource::Group with full_path 'gitlab-qa-sandbox-group-6/qa-test-2024-04-12-22-57-22-3b94cf102797777d' via api in 0.62 seconds
[Apr 12 2024 23:04:33 UTC (QA Tests)] INFO -- ==> Built a QA::Resource::Project with full_path 'gitlab-qa-sandbox-group-6/qa-test-2024-04-12-22-57-22-3b94cf102797777d/file_locking-495625b54a9215e9' via api in 3.65 seconds
Terminated
[Apr 12 2024 21:00:27 UTC (QA Tests)] INFO -- Starting test: Create File Locking creates a merge request and fails to merge
[Apr 12 2024 21:00:28 UTC (QA Tests)] INFO -- filling :username-field with "gitlab-qa"
[Apr 12 2024 21:00:28 UTC (QA Tests)] INFO -- clicking the coordinates of :username-field
[Apr 12 2024 21:00:28 UTC (QA Tests)] INFO -- filling :password-field with "*****"
[Apr 12 2024 21:00:28 UTC (QA Tests)] INFO -- clicking the coordinates of :password-field
[Apr 12 2024 21:00:29 UTC (QA Tests)] INFO -- clicking :sign-in-button
[Apr 12 2024 21:00:31 UTC (QA Tests)] WARN -- Potentially Slow Code 'click_element sign-in-button' took 2.539s
[Apr 12 2024 21:00:34 UTC (QA Tests)] INFO -- ==> Retrieved a QA::Resource::Sandbox with full_path 'gitlab-qa-sandbox-group-6' via api in 0.21 seconds
[Apr 12 2024 21:00:35 UTC (QA Tests)] INFO -- ==> Retrieved a QA::Resource::Group with full_path 'gitlab-qa-sandbox-group-6/qa-test-2024-04-12-20-38-06-cab3a71830e9a5f1' via api in 0.63 seconds
[Apr 12 2024 21:00:38 UTC (QA Tests)] INFO -- ==> Built a QA::Resource::Project with full_path 'gitlab-qa-sandbox-group-6/qa-test-2024-04-12-20-38-06-cab3a71830e9a5f1/file_locking-c061f23926b7afb8' via api in 4.41 seconds
[Apr 12 2024 21:00:41 UTC (QA Tests)] INFO -- QA::Resource::Project - wait_for_push with commit message "This is a test commit"
[Apr 12 2024 21:00:41 UTC (QA Tests)] INFO -- ==> Built a QA::Resource::Repository::ProjectPush via browser_ui in 3.22 seconds
[Apr 12 2024 21:00:42 UTC (QA Tests)] INFO -- ==> Retrieved a QA::Resource::User with username 'gitlab-qa-user1' via api in 0.43 seconds
[Apr 12 2024 21:00:43 UTC (QA Tests)] INFO -- ==> Built a QA::Resource::ProjectMember via api in 0.77 seconds
[Apr 12 2024 21:00:43 UTC (QA Tests)] INFO -- ==> Retrieved a QA::Resource::User with username 'gitlab-qa-user2' via api in 0.38 seconds
[Apr 12 2024 21:00:44 UTC (QA Tests)] INFO -- ==> Built a QA::Resource::ProjectMember via api in 1.23 seconds
[Apr 12 2024 21:00:46 UTC (QA Tests)] INFO -- Removed a QA::Resource::ProtectedBranch
[Apr 12 2024 21:00:46 UTC (QA Tests)] WARN -- Resource fabrication http method has not been set properly, assuming :get value!
[Apr 12 2024 21:00:46 UTC (QA Tests)] WARN -- Resource fabrication http method has not been set properly, assuming :get value!
[Apr 12 2024 21:00:46 UTC (QA Tests)] INFO -- ==> Built a QA::Resource::ProtectedBranch via api in 1.21 seconds
Terminated
Screenshot / HTML page
Possible fixes
- For
Create Push Rules using non signed commits restricts files by name and size
, considering the test is simply checking that an error occurs when attempting to push either a large file or a wrongly named file, it looks like the large file creationSecureRandom.hex(1000000)
on line 45 might be timing out here. Could we consider using a test fixture that is already 1 MiB instead? - Looks like
Create File Locking locks a file and tries to push as a second user
, also has a step where it is callingSecureRandom.hex(1000000)
and pushing that file into the repository, that might be where it's timing out at times when the test terminates atBuilt a QA::Resource::ProtectedBranch via api
. As for the one time the test went all the way to theclick_lock
step, I'm not very familiar with how locking works but is it potentially timing out due to the "large" file? Wondering if it was necessary to create a file of this size for this test. Maybe could reduce it and see if the timeouts continue to happen? - For
Create File Locking creates a merge request and fails to merge
, it seems to have some similarity with the failure inCreate Push Rules using non signed commits restricts files by name and size
so it could be the same issue as well.