WIP: Use high CPU runners for assets compilation jobs
What does this MR do?
Experiment to use high CPU runners for assets compilation jobs, suggested in https://gitlab.com/gitlab-org/gitlab-ce/issues/56538#note_192485436.
That looks promising:
Job | master |
this MR |
---|---|---|
compile-assets rake:assets:precompile
|
82613ms , 82945ms , 82281ms
|
65527ms |
compile-assets total |
11m 10s , 8m 39s , 8m 45s
|
6m 26s |
gitlab:assets:compile rake:assets:precompile
|
479239ms , 520504ms , 588495ms
|
500880ms |
gitlab:assets:compile |
19m 3s , 19m 34s , 21m 13s
|
17m 7s |
Merge request reports
Activity
changed milestone to %12.2
added Quality backstage [DEPRECATED] ci-build labels
Reviewer roulette
Changes that require review have been detected! A merge request is normally reviewed by both a reviewer and a maintainer in its primary category (e.g. frontend or backend), and by a maintainer in all other categories.
These files couldn't be categorised, so Danger was unable to suggest a reviewer. Please consider creating a merge request to add support for them.
.gitlab/ci/frontend.gitlab-ci.yml
Generated by
DangerEdited by 🤖 GitLab Bot 🤖added 1 commit
- 4d0e29f9 - Use high CPU runners for assets compilation jobs
/cc FYI @gl-quality/eng-prod that looks like an easy win.
I agree the results look like an improvement and am trying to think if there's any additional testing or considerations we should take.
Changing the CE Pipeline to these runners would increase the load that hits these runners by a large factor. I can't seem to find any documentation in the docs on how many
high-cpu
runners are available. Do you know where can I find out how many runners are in this group and if it autoscales? I was hunting through the docs and am thinking I'm overlooking this.@kwiebers Good point. We can ask @tmaczukin about these runners?
On GitLab.com we have 4 runners managers (referenced as
gsrmX
-gitlab-shared-runners-manager-X.gitlab.com
) configured withn1-standard-2
GCP machine. We have also 2 other runner managers (referenced asprmX
-private-runners-manager-X.gitlab.com
) that use the same configuration. Each of them is configured to autoscale up to 400 (forgsrmX
) and up to 300 (forprmX
) machines per manager.For forks only the
gsrmX
are available, since theprmX
managers are added as Group Runners ongitlab-org
group level.The same machines are also available as Shared Runners on dev.gitlab.org. For dev, the machines should scale up to ~250 (for
gsrmX
) and ~200 (forprmX
).Please also note, that for
gsrmX
managers there is a top-level concurrency limit set to 500 jobs per manager. Workers for GitLab.com and dev.gitlab.org are sharing this limit.For
prmX
managers, there is a global concurrency set to 400 jobs per manager, and this is shared betweengitlab-org
,gitlab-com
,charts
groups at GitLab.com and dev.gitlab.org.
16 16 <<: *assets-compile-cache 17 17 extends: .dedicated-no-docs-pull-cache-job 18 18 image: dev.gitlab.org:5005/gitlab/gitlab-build-images:ruby-2.6.3-git-2.21-chrome-73.0-node-12.x-yarn-1.16-graphicsmagick-1.3.29-docker-18.06.1 19 tags: 20 - gitlab-org 21 - docker 22 - high-cpu We have no Shared Runners on GitLab.com tagged with
high-cpu
, only a private ones added togitlab-org
group.This will make this job stuck for any contributor that uses own fork instead of https://gitlab.com/gitlab-org/gitlab-ce or https://gitlab.com/gitlab-org/gitlab-ee
@tmaczukin Is there any way to fallback to non-high-cpu runners in that case (I don't think so, but I prefer to ask in any case)?
@rymai No. There is no "fallback" mechanism in job tags checking.
But
gitlab-org
is already assigned to the proper Runner Managers. I don't see a reason to add another tag here :)And of course (if this is not clear) there are 4
gitlab-org
tagged Shared Runners on GitLab.com. So adding onlygitlab-org
will make it available for forks.However, we have no
gitlab-shared-runners-manager-X
Runner with adocker
tag. It's because the managers are not configured with privileged mode set to true and will not allow to build Docker images (security concerns; machines are re-used between multiple jobs). So for jobs that require Docker building, we need to stay with the regular, non-gitlab-org
-tagged Shared Runners to make it working.But
gitlab-org
is already assigned to the proper Runner Managers. I don't see a reason to add another tag here :)@tmaczukin Does that mean adding the
high-cpu
tag here has no impact at all on the performance of the jobs I added them to? In that case, maybe thehigh-cpu
tag is misleading?However, we have no
gitlab-shared-runners-manager-X
Runner with adocker
tag. It's because the managers are not configured with privileged mode set to true and will not allow to build Docker images (security concerns; machines are re-used between multiple jobs). So for jobs that require Docker building, we need to stay with the regular, non-gitlab-org
-tagged Shared Runners to make it working.I see, that makes sense.
@tmaczukin I see we have https://docs.gitlab.com/ee/user/gitlab_com/#shared-runners that describe the GitLab.com shared runners, but do we have the same for the
gitlab-org
runners, by any chance?Does that mean adding the
high-cpu
tag here has no impact at all on the performance of the jobs I added them to? In that case, maybe thehigh-cpu
tag is misleading?Anyone that have maintainer access to
gitlab-org
group or admin access to dev.gitlab.org (because these are the places where Runners are tagged withhigh-cpu
) could add such tag. Don't ask me why From scheduling point of view it's enough that these Runners are available for GitLab CE project and havegitlab-org
label.And yes, with Runners available in GitLab CE project, adding
high-cpu
to jobs withgitlab-org
tag doesn't change anything.I see we have https://docs.gitlab.com/ee/user/gitlab_com/#shared-runners that describe the GitLab.com shared runners, but do we have the same for the
gitlab-org
runners, by any chance?AFAIR, it's not documented in any place, but you can find details in the chef-repo (all Shared Runners are for now managed by Chef)
AFAIR, it's not documented in any place, but you can find details in the chef-repo (all Shared Runners are for now managed by Chef)
@tmaczukin Where can we find this chef-repo project?
@rymai The configuration is composed from these roles (in the order of details specification):
- https://ops.gitlab.net/gitlab-cookbooks/chef-repo/blob/master/roles/gitlab-runner-base.json
- https://ops.gitlab.net/gitlab-cookbooks/chef-repo/blob/master/roles/gitlab-runner-gsrm.json
- https://ops.gitlab.net/gitlab-cookbooks/chef-repo/blob/master/roles/gitlab-runner-gsrm-gce.json
- one of: https://ops.gitlab.net/gitlab-cookbooks/chef-repo/blob/master/roles/gitlab-runner-gsrm-gce-us-east1-c.json or https://ops.gitlab.net/gitlab-cookbooks/chef-repo/blob/master/roles/gitlab-runner-gsrm-gce-us-east1-d.json (depending on the zone in which the Runner manager is deployed)
@tmaczukin Awesome, thanks for the links! Does the Verify team owns the runner managers management? If yes, what do you think of documenting that somewhere under https://about.gitlab.com/handbook/engineering/development/ci-cd/verify/ in a smilar way it's done in https://about.gitlab.com/handbook/engineering/development/enablement/distribution/maintenance/?
mentioned in issue gitlab#34319 (closed)