Skip to content
Snippets Groups Projects

WIP: Use high CPU runners for assets compilation jobs

Closed Rémy Coutable requested to merge use-high-cpu-machines-for-assets-compilation into master
3 unresolved threads

What does this MR do?

Experiment to use high CPU runners for assets compilation jobs, suggested in https://gitlab.com/gitlab-org/gitlab-ce/issues/56538#note_192485436.

That looks promising:

Job master this MR
compile-assets rake:assets:precompile 82613ms, 82945ms, 82281ms 65527ms
compile-assets total 11m 10s, 8m 39s, 8m 45s 6m 26s
gitlab:assets:compile rake:assets:precompile 479239ms, 520504ms, 588495ms 500880ms
gitlab:assets:compile 19m 3s, 19m 34s, 21m 13s 17m 7s
Edited by Rémy Coutable

Merge request reports

Loading
Loading

Activity

Filter activity
  • Approvals
  • Assignees & reviewers
  • Comments (from bots)
  • Comments (from users)
  • Commits & branches
  • Edits
  • Labels
  • Lock status
  • Mentions
  • Merge request status
  • Tracking
  • Rémy Coutable marked as a Work In Progress

    marked as a Work In Progress

  • Rémy Coutable changed the description

    changed the description

  • Rémy Coutable changed milestone to %12.2

    changed milestone to %12.2

  • Reviewer roulette

    Changes that require review have been detected! A merge request is normally reviewed by both a reviewer and a maintainer in its primary category (e.g. frontend or backend), and by a maintainer in all other categories.

    These files couldn't be categorised, so Danger was unable to suggest a reviewer. Please consider creating a merge request to add support for them.

    • .gitlab/ci/frontend.gitlab-ci.yml

    Generated by :no_entry_sign: Danger

    Edited by 🤖 GitLab Bot 🤖
  • Rémy Coutable added 1 commit

    added 1 commit

    • 4d0e29f9 - Use high CPU runners for assets compilation jobs

    Compare with previous version

  • Rémy Coutable changed the description

    changed the description

  • Author Maintainer

    /cc FYI @gl-quality/eng-prod that looks like an easy win. :thinking:

    • I agree the results look like an improvement and am trying to think if there's any additional testing or considerations we should take.

      Changing the CE Pipeline to these runners would increase the load that hits these runners by a large factor. I can't seem to find any documentation in the docs on how many high-cpu runners are available. Do you know where can I find out how many runners are in this group and if it autoscales? I was hunting through the docs and am thinking I'm overlooking this.

    • Author Maintainer

      @kwiebers Good point. We can ask @tmaczukin about these runners?

    • On GitLab.com we have 4 runners managers (referenced as gsrmX - gitlab-shared-runners-manager-X.gitlab.com) configured with n1-standard-2 GCP machine. We have also 2 other runner managers (referenced as prmX - private-runners-manager-X.gitlab.com) that use the same configuration. Each of them is configured to autoscale up to 400 (for gsrmX) and up to 300 (for prmX) machines per manager.

      For forks only the gsrmX are available, since the prmX managers are added as Group Runners on gitlab-org group level.

      The same machines are also available as Shared Runners on dev.gitlab.org. For dev, the machines should scale up to ~250 (for gsrmX) and ~200 (for prmX).

      Please also note, that for gsrmX managers there is a top-level concurrency limit set to 500 jobs per manager. Workers for GitLab.com and dev.gitlab.org are sharing this limit.

      For prmX managers, there is a global concurrency set to 400 jobs per manager, and this is shared between gitlab-org, gitlab-com, charts groups at GitLab.com and dev.gitlab.org.

    • Please register or sign in to reply
16 16 <<: *assets-compile-cache
17 17 extends: .dedicated-no-docs-pull-cache-job
18 18 image: dev.gitlab.org:5005/gitlab/gitlab-build-images:ruby-2.6.3-git-2.21-chrome-73.0-node-12.x-yarn-1.16-graphicsmagick-1.3.29-docker-18.06.1
19 tags:
20 - gitlab-org
21 - docker
22 - high-cpu
  • 79 80 extends: .dedicated-runner
    80 81 <<: *use-pg
    81 82 <<: *assets-compile-cache
    83 tags:
    84 - gitlab-org
    85 - high-cpu
  • Author Maintainer

    Closing as the high-cpu tag has no impact at all!

  • Please register or sign in to reply
    Loading