Improve reliability of Packagecloud CI jobs
We had an issue where some but not all Runner packages were published to Packagecloud in #36048 (comment 1492541219).
It seems that the ci/push_packagecloud
script doesn't fail upon error because this job is run multiple times, and we want to ensure that the existence of a package doesn't abort the process since publishing should be idempotent.
For example:
-
https://gitlab.com/gitlab-org/gitlab-runner/-/jobs/4517690577 (
stable packagecloud: [debian]
) -
https://gitlab.com/gitlab-org/gitlab-runner/-/jobs/4517690578 (
stable packagecloud: [ubuntu]
)
Both jobs publish .deb
files for the both Ubuntu and Debian, so there is a bit of redundancy here. However, we can't simply add set -eo pipefail
to the script because we assume the existence of a file is NOT a hard error. We have many messages like:
Pushing out/deb/gitlab-runner_amd64.deb... error:
filename: has already been taken
Some ideas for improvement here:
-
Update the script so that the presence of the file does not cause an exit error. It looks like the
package_cloud
gem performs a hard exit upon error unless the--skip-errors
parameter is passed:def create_package(client_repo, f, dist_id, files, ext, coordinates=nil) begin client_repo.create_package(f, dist_id, files, ext, coordinates) rescue RestClient::UnprocessableEntity => e print "error:\n".color(:red) json = JSON.parse(e.response) json.each do |k,v| if v.is_a? String puts "\n\t#{k}: #{v}\n" elsif v.is_a? Array puts "\n\t#{k}: #{v.join(", ")}\n" end end puts "" exit(1) end
-
Consolidate the CI jobs for publishing packages so that there are only 1 or 2 jobs for Packagecloud sync (Debian and RPM packages).
-
Update the
ci/push_packagecloud
script to useset -eo pipefail
. -
Add another CI job to check that all packages were published (?).