Skip to content

Improve build performance

Darby Frey requested to merge build-timing into main

Why is this change being made?

I looked at a few things in this MR to evaluate our bottlenecks in the handbook CI pipeline, below is a summary of findings and proposed changes.

Note: these stats are for the build and test jobs. The deploy jobs were not included in this analysis.

Looking at https://gitlab.com/gitlab-com/content-sites/handbook/-/pipelines/1301490248/builds, it shows there are 5 build/test in the pipeline. The table below shows jobs and timing. The build job takes about 3x longer than the rest.

Job Inital Time
build 8:10
markdownlint 2:31
handbooklint 1:12
hugolint 1:05
secret_detection 0:59

I added some collapsible groups to the build script to get sense of where the time was going. See the job log here: https://gitlab.com/gitlab-com/content-sites/handbook/-/jobs/6916843233. From that we can see that the most time is spent in the hugo build, a find/sed, and artifact upload.

We should look for ways to tune those three commands to improve build performance.

The other thing I considered was the machine size. I ran a test on our default machine and two larger machine types. The medium machine type saw a nice improvement, but performance seemed to level off there and the large machine only saw a small improvement. See the table below:

Machine Type slowest job hugo sed artifact upload total time
saas-linux-small-amd64 build (8:10) 3:04 1:45 2:06 8:10
saas-linux-medium-amd64 build (5:26) 1:30 1:09 1:40 5:26
saas-linux-large-amd64 build (5:10) 0:52 1:32 1:38 5:10

This MR is proposing using the saas-linux-medium-amd64 machine for these jobs as well as moving all of the build and test jobs into one stage so they can all run in parallel. There are no dependences between the stages, so we can save time by running them all together.

Update

Added the suggestions here to improve the artifact upload performance. This change cuts about a minute off of the artifact upload step.

Machine Type slowest job hugo sed artifact upload total time
saas-linux-medium-amd64 build (4:26) 1:31 1:06 0:40 4:26

Author and Reviewer Checklist

Please verify the check list and ensure to tick them off before the MR is merged.

  • Provided a concise title for this Merge Request (MR)
  • Added a description to this MR explaining the reasons for the proposed change, per say why, not just what
    • Copy/paste the Slack conversation to document it for later, or upload screenshots. Verify that no confidential data is added, and the content is SAFE
  • Assign reviewers for this MR to the correct Directly Responsible Individual/s (DRI)
    • If the DRI for the page/s being updated isn’t immediately clear, then assign it to one of the people listed in the Maintained by section on the page being edited
    • If your manager does not have merge rights, please ask someone to merge it AFTER it has been approved by your manager in #mr-buddies
    • The when to get approval handbook section explains the workflow in more detail
  • For transparency, share this MR with the audience that will be impacted.
    • Team: For changes that affect your direct team, share in your group Slack channel
    • Department: If the update affects your department, share the MR in your department Slack channel
    • Division: If the update affects your division, share the MR in your division Slack channel
    • Company: If the update affects all (or the majority of) GitLab team members, post an update in #whats-happening-at-gitlab linking to this MR

Commits

  • Added timing info to build job
Edited by Darby Frey

Merge request reports