Limit concurrency of package publish jobs using resource_group
What does this MR do?
The package publish jobs can easily overwhelm the package server. When all the publish jobs are executed at the same time, most of them fail due to 5xx responses from the package server.
This commit uses a resource_group
to limit the number of concurrent
package publish jobs that can be executed. They will be executed one
at a time per commit. So, if there are 6 different pipelines running on 6 different commits, you can have 6 jobs running concurrently, 1 per pipeline.
During a security release, the publish jobs in 6 pipelines are triggered simultaneously. Each package publish job takes 2-4 minutes to complete as can be seen in https://dev.gitlab.org/gitlab/omnibus-gitlab/-/pipelines/246266/builds. Excluding the docker and ami jobs (since they don't hit the package server), there are about 15 publish jobs per pipeline. Assuming 3 minutes on average per job, 15*3 == 45 minutes
. So it should take 45 minutes for each pipeline's package publish jobs to complete.
Example pipeline where multiple package publish jobs failed and had to be manually retried: https://dev.gitlab.org/gitlab/omnibus-gitlab/-/pipelines/246218.
Test pipeline using a resource_group
containing CI_COMMIT_SHA
variable: https://gitlab.com/rpereira2/release-tools-fake/-/pipelines/604069046.
Related issues
Checklist
See Definition of done.
For anything in this list which will not be completed, please provide a reason in the MR discussion
Required
-
Merge Request Title, and Description are up to date, accurate, and descriptive -
MR targeting the appropriate branch -
MR has a green pipeline on GitLab.com -
Pipeline is green on dev.gitlab.org if the change is touching anything besides documentation or internal cookbooks -
trigger-package
has a green pipeline running against latest commit
Expected (please provide an explanation if not completing)
Not completing these since this MR is a minor change to the CI pipeline.
-
Test plan indicating conditions for success has been posted and passes -
Documentation created/updated -
Tests added -
Integration tests added to GitLab QA -
Equivalent MR/issue for the GitLab Chart opened