Skip to content

GitLab Next

  • Projects
  • Groups
  • Snippets
  • Help
    • Loading...
  • Help
    • Help
    • Support
    • Community forum
    • Submit feedback
    • Contribute to GitLab
  • Sign in / Register
gitlab-runner
gitlab-runner
  • Project overview
    • Project overview
    • Details
    • Activity
    • Releases
  • Repository
    • Repository
    • Files
    • Commits
    • Branches
    • Tags
    • Contributors
    • Graph
    • Compare
    • Locked Files
  • Issues 2,041
    • Issues 2,041
    • List
    • Boards
    • Labels
    • Service Desk
    • Milestones
    • Iterations
  • Merge Requests 203
    • Merge Requests 203
  • Requirements
    • Requirements
    • List
  • CI / CD
    • CI / CD
    • Pipelines
    • Jobs
    • Schedules
    • Test Cases
  • Operations
    • Operations
    • Incidents
    • Environments
  • Packages & Registries
    • Packages & Registries
    • Container Registry
  • Analytics
    • Analytics
    • CI / CD
    • Code Review
    • Insights
    • Issue
    • Repository
    • Value Stream
  • Members
    • Members
  • Activity
  • Graph
  • Create a new issue
  • Jobs
  • Commits
  • Issue Boards
Collapse sidebar
  • GitLab.org
  • gitlab-runnergitlab-runner
  • Merge Requests
  • !2358

Merged
Opened Aug 19, 2020 by Erik Lindahl@erik.lindahlContributor

Improve cache upload speed

  • Overview 8
  • Commits 1
  • Pipelines 2
  • Changes 1

What does this MR do?

It makes cache uploading 10x faster on high-speed networks.

Why was this MR needed?

The default net/http setup of enforcing compression in the HTTP transport leads to huge slowdowns for large file transfers. For this reason, other common Go packages (such as the Minio tools) also disable transport-layer compression this way.

What's the best way to test this MR?

The HTTP settings themselves are completely opaque to the rest of the code, it's just a setup option. The best way to confirm the speed improvement is to test a large cache file (~1GB) from a runner directly connected to an S3 cache repo through a 10/100 Gb network.

Another way of trusting it could arguably be that this change means GitLab-runner will use the same settings as Minio and other tools that have received a lot performance tuning and testing specifically for upload/download bandwidth.

What are the relevant issue numbers?

closes #25801 (closed)

Edited Nov 20, 2020 by Steve Azzopardi
Assignee
Assign to
Reviewer
Request review from
13.7
Milestone
13.7 (Past due)
Assign milestone
Time tracking
Reference: gitlab-org/gitlab-runner!2358
Source branch: improve_cache_upload_perf