S3 Object Storage Artifact Upload Speed is slower than download speed

Issue Summary

Customer in https://gitlab.zendesk.com/agent/tickets/612186 (internal use only) is reporting the following discrepancy with their artifact uploads:

  • Download Speed: Up to 300+ MB/s
  • Upload Speed: Limited to 17-30 MB/s
  • Environment: S3 object storage with 10GB local network and 5GB internet connection
  • Testing: With rclone they see 88.457 MiB/s when they upload the same file

Root Cause Analysis

After thorough investigation, we've identified that the speed discrepancy is due to fundamental architectural differences in how GitLab handles downloads versus uploads:

graph TD
    subgraph "Download Flow (Fast: 300+ MB/s)"
        A[CI Runner] -->|"Direct S3 Access via Pre-signed URL"| B[S3 Object Storage]
    end
    
    subgraph "Upload Flow (Slow: 17-30 MB/s)"
        C[CI Runner] -->|"HTTP Upload"| D[GitLab Server]
        D -->|"Workhorse Processing"| E[Workhorse]
        E -->|"Single-threaded Upload"| F[S3 Object Storage]
    end

Download Path

  • Uses pre-signed URLs allowing runners to download directly from S3
  • Completely bypasses the GitLab server
  • Results in optimal performance (300+ MB/s)

Upload Path

  • Data must flow through GitLab Workhorse before reaching S3
  • This additional network hop creates a bottleneck
  • Limits speeds to 17-30 MB/s regardless of network capacity

Technical Details

  • This is not a runner limitation but a Workhorse architectural constraint
  • Unlike cache uploads (which can now upload directly to S3), artifacts must be processed by the GitLab instance
  • The issue is not specifically about multi-part uploads, though that would help
  • The fundamental problem is the required Workhorse intermediary step for artifact uploads

Recommendation

  1. Implementing direct-to-S3 uploads for artifacts (similar to how downloads work)
  2. Bypassing the GitLab server for artifact uploads when using object storage
  3. Supporting parallel upload streams to maximize bandwidth utilization

Related Issues

Edited by 🤖 GitLab Bot 🤖