Jenkins spec: add check for the latest job build status in Jenkins logs
What does this MR do?
Related to #195179 (closed)
- Failure 1:
expected to find css ".js-pipeline-url-link" at least 1 time but there were no matches- example https://gitlab.com/gitlab-org/gitlab-qa/-/jobs/444254692
Looking through all the error logs for Jenkins spec it seems that this error happens the most of the time. According to the spec code - we're pushing commit to the project, it triggers Jenkins CI build, we check that the CI build was successful, then we go to our GitLab instance and check that we see a build in Pipelines - it happens within 3 retries, so if we don't see for some reason build in GitLab Pipelines, we push a new commit all over again. I wasn't able to reproduce this issue locally(ran it multiple times), also it passed successfully 3 times against the Nightly, so the problem is that the issue is very flaky.
Solution:
- Check for job build status inside console output: wait for build to success + ensure that we don't see
Failed to update Gitlab commit statusin the logs - Thanks Andy! !27695 (closed) - It tries to check for failures in updating the GitLab commit status on Jenkins using the job console. Unfortunately there is no console message when the status update was successful. - Spec was refactored a bit in accordance to the addition above.
- Failure 2:
unexpected alert open: {Alert text : }- example https://gitlab.com/gitlab-org/gitlab-qa/-/jobs/436958042
Was able to reproduce this error when removed click_save from configure_job.rb - since the failure is flaky, the assumption here is that Job Configuration is not yet saved and we already try to login to GitLab, which leads to this error, because Jenkins pops up Chrome standard alert Reload the site? and it blocks Capybara to load GitLab login page.
Solution:
- Wait for Job Configuration to save. When Configuration is save we're being redirected to Project main page.
Proposal
Merge this MR and keep an eye on spec results in quarantined Nightly job. If it passes for 5 times, let's unquarantine the test and close the issue above.
If the spec still fail, we can enhance Jenkins logs to see what happened on Jenkins part. We can add Jenkins Logs page and the ability to wait for gitlabjenkins.util.CommitStatusUpdater updateCommitStatus in the logs that tells us that at least Jenkins sent build status data to GitLab instance and if we still won't see pipelines in GitLab we should be sure that something wrong on GitLab side. Another solution would be to use only API requests for Jenkins setup part to make it more robust.
Screenshots
- Failure 1:
expected to find css ".js-pipeline-url-link" at least 1 time but there were no matches- Example of the Jenkins Job console logs - if the request to GitLab fails, we see it in the logs like this:
- Failure 2:
unexpected alert open: {Alert text : }- standard Chrome alert if we try to leave the page before new Job Configuration is saved:
