Skip to content

GitLab Next

  • Projects
  • Groups
  • Snippets
  • Help
    • Loading...
  • Help
    • Help
    • Support
    • Community forum
    • Submit feedback
    • Contribute to GitLab
  • Sign in / Register
gitlab-runner
gitlab-runner
  • Project overview
    • Project overview
    • Details
    • Activity
    • Releases
  • Repository
    • Repository
    • Files
    • Commits
    • Branches
    • Tags
    • Contributors
    • Graph
    • Compare
    • Locked Files
  • Issues 1,996
    • Issues 1,996
    • List
    • Boards
    • Labels
    • Service Desk
    • Milestones
    • Iterations
  • Merge Requests 204
    • Merge Requests 204
  • Requirements
    • Requirements
    • List
  • CI / CD
    • CI / CD
    • Pipelines
    • Jobs
    • Schedules
    • Test Cases
  • Security & Compliance
    • Security & Compliance
    • Dependency List
    • License Compliance
  • Operations
    • Operations
    • Incidents
    • Environments
  • Packages & Registries
    • Packages & Registries
    • Container Registry
  • Analytics
    • Analytics
    • CI / CD
    • Code Review
    • Insights
    • Issue
    • Repository
    • Value Stream
  • Members
    • Members
  • Collapse sidebar
  • Activity
  • Graph
  • Create a new issue
  • Jobs
  • Commits
  • Issue Boards
  • GitLab.org
  • gitlab-runnergitlab-runner
  • Issues
  • #25801

Closed
Open
Opened May 22, 2020 by Erik Lindahl@erik.lindahlContributor

Improve cache upload speed on high-speed networks

Release notes

  • The related merge request that addresses this issue disables transport-layer compression in gitlab-runner cache_client.go

Summary

Well, perhaps not a strict bug, but definitely a major performance degradation that seems unnecessary :-)

I have been trying to diagnose slow cache operations, and after hacking the gitlab-runner-helper binaries with more timing information I found that the download part of the cache_extractor operations can be extremely slow. This appears to be do to a combination of small buffers and the Golang built-in version of http not being optimized for bandwidth.

Steps to reproduce

  • Add timing operations around the io.Copy() operation in routine download() of commands/helpers/cache_extractor.go.
  • Use a local S3 server, without SSL (to enable highest-possible performance).
  • When downloading a 931MB cache file with the 'mc' client outside of gitlab-runner, it takes 1.9seconds (10Gb network), so network and S3 server setup is A-OK.

Actual behavior

  • Gitlab-runner takes 42 seconds to download the same file.

Expected behavior

  • When I hack the code to use the Minio Golang package instead of presigned URLs + HTTP, the download takes 1.9 seconds in gitlab-runner too.

Environment description

  • Using our own runners in a k8s cluster
  • Ceph-backed radosgw S3 storage in the same cluster
  • No SSL to improve bandwidth

Used GitLab Runner version

13.1.0 (development)

Possible fixes

If there's interest, we could probably provide a MR to use Minio instead, although there could be some extra complications to handle GCS natively too.

Edited Dec 09, 2020 by Darren Eastman
Assignee
Assign to
13.7
Milestone
13.7 (Past due)
Assign milestone
Time tracking
None
Due date
None
Reference: gitlab-org/gitlab-runner#25801