Internal release: Release job failed due to commit not being available on dev
The job failed because the commit was not available on dev https://ops.gitlab.net/gitlab-org/release/tools/-/jobs/17014892:
2025-01-31 15:55:33.414689 I ReleaseTools::InternalRelease::Release -- Waiting for commit in dev mirror
2025-01-31 15:55:33.510140 D ReleaseTools::GitlabDevClient -- [HTTParty] [2025-01-31 15:55:33 +0000] 404 "GET https://dev.gitlab.org/api/v4/projects/gitlab%2Fgitlab-ee/repository/commits/bfd70cbed3ccb833cfacc2585a3a60edd26edd55" 34
2025-01-31 15:55:33.510413 F ReleaseTools::InternalRelease::Release -- Building the internal release package 17.7.3-internal0 failed with error: Server responded with code 404, message: 404 Commit Not Found. Request URI: https://dev.gitlab.org/api/v4/projects/gitlab%2Fgitlab-ee/repository/commits/bfd70cbed3ccb833cfacc2585a3a60edd26edd55.
Review the error log and consider retrying this job.
-- {error: #<Gitlab::Error::NotFound: Server responded with code 404, message: 404 Commit Not Found. Request URI: https://dev.gitlab.org/api/v4/projects/gitlab%2Fgitlab-ee/repository/commits/bfd70cbed3ccb833cfacc2585a3a60edd26edd55>}
2025-01-31 15:55:33.510624 I ReleaseTools::Slack::ReleaseJobEndNotifier -- Posting slack message -- {job_type: "Build internal release package 17.7.3-internal0", status: :failed}
It failed with the same error after a retry https://ops.gitlab.net/gitlab-org/release/tools/-/jobs/17033471
Looks like there was a small delay in propagating the commits to dev which caused the failure:
I've a couple of questions/observations regarding this:
- Looks like the Retriable block isn't working
- Why do we need to wait for the commit to be on dev?
- There is also another job to wait for the
build_assets
on dev. I suspect it may take a while for that job to finish, so we should analyze if it is needed.
To do
-
Fix and simplify the internal release logic service gitlab-org/release-tools!3852 (merged) -
Add the QA waiting bit under a feature flag gitlab-org/release-tools!3880 (merged) -
Ensure Security and Dev are running the same commit before triggering pipelines gitlab-org/release-tools!3909 (merged) -
Increase the job timeout to 1h. gitlab-org/release-tools!3908 (merged) -
Test on a fake stable branch #20859 (comment 2359375476)
Follow-ups
Edited by Mayra Cabrera