Skip to content

Limit concurrency of package publish jobs using resource_group

Reuben Pereira requested to merge rp/limit-publish-jobs-concurrency into master

What does this MR do?

The package publish jobs can easily overwhelm the package server. When all the publish jobs are executed at the same time, most of them fail due to 5xx responses from the package server.

This commit uses a resource_group to limit the number of concurrent package publish jobs that can be executed. They will be executed one at a time per commit. So, if there are 6 different pipelines running on 6 different commits, you can have 6 jobs running concurrently, 1 per pipeline.

During a security release, the publish jobs in 6 pipelines are triggered simultaneously. Each package publish job takes 2-4 minutes to complete as can be seen in https://dev.gitlab.org/gitlab/omnibus-gitlab/-/pipelines/246266/builds. Excluding the docker and ami jobs (since they don't hit the package server), there are about 15 publish jobs per pipeline. Assuming 3 minutes on average per job, 15*3 == 45 minutes. So it should take 45 minutes for each pipeline's package publish jobs to complete.

Example pipeline where multiple package publish jobs failed and had to be manually retried: https://dev.gitlab.org/gitlab/omnibus-gitlab/-/pipelines/246218.

Test pipeline using a resource_group containing CI_COMMIT_SHA variable: https://gitlab.com/rpereira2/release-tools-fake/-/pipelines/604069046.

Related issues

Checklist

See Definition of done.

For anything in this list which will not be completed, please provide a reason in the MR discussion

Required

  • Merge Request Title, and Description are up to date, accurate, and descriptive
  • MR targeting the appropriate branch
  • MR has a green pipeline on GitLab.com
  • Pipeline is green on dev.gitlab.org if the change is touching anything besides documentation or internal cookbooks
  • trigger-package has a green pipeline running against latest commit

Expected (please provide an explanation if not completing)

Not completing these since this MR is a minor change to the CI pipeline.

  • Test plan indicating conditions for success has been posted and passes
  • Documentation created/updated
  • Tests added
  • Integration tests added to GitLab QA
  • Equivalent MR/issue for the GitLab Chart opened
Edited by Reuben Pereira

Merge request reports