Skip to content
GitLab
Next
Projects Groups Snippets
  • /
  • Help
    • Help
    • Support
    • Community forum
    • Submit feedback
    • Contribute to GitLab
  • Sign in / Register
  • gitlab-workhorse gitlab-workhorse
  • Project information
    • Project information
    • Activity
    • Labels
    • Members
  • Repository
    • Repository
    • Files
    • Commits
    • Branches
    • Tags
    • Contributors
    • Graph
    • Compare
    • Locked Files
  • Issues 0
    • Issues 0
    • List
    • Boards
    • Service Desk
    • Milestones
    • Iterations
    • Requirements
  • Merge requests 0
    • Merge requests 0
  • CI/CD
    • CI/CD
    • Pipelines
    • Jobs
    • Schedules
    • Test Cases
  • Deployments
    • Deployments
    • Environments
    • Releases
  • Monitor
    • Monitor
    • Incidents
  • Analytics
    • Analytics
    • Value stream
    • CI/CD
    • Code review
    • Insights
    • Issue
    • Repository
  • Activity
  • Graph
  • Create a new issue
  • Jobs
  • Commits
  • Issue Boards
Collapse sidebar
  • GitLab.orgGitLab.org
  • gitlab-workhorsegitlab-workhorse
  • Merge requests
  • !466

Support Workhorse directly uploading files to S3 [3/3]

  • Review changes

  • Download
  • Email patches
  • Plain diff
Merged Stan Hu requested to merge sh-workhorse-direct-s3 into master Feb 23, 2020
  • Overview 131
  • Commits 1
  • Pipelines 100
  • Changes 13

This adds the AWS client directly to Workhorse and a new configuration section for specifying credentials. This makes it possible to use S3 buckets with KMS encryption and proper MD5 checksums.

This is disabled by default. For this to be used:

  1. GitLab Rails needs to send the UseWorkhorseClient and RemoteTempObjectID in the /authorize endpoint. (gitlab!29389 (merged))

  2. S3 configuration must be specified in config.toml, or Rails must be configured to use IAM instance profiles (use_iam_profile in Fog connection parameters).

S3 sessions are created lazily and cached for 10 minutes to avoid unnecessary local I/O access. When IAM instance profiles are used, this also cuts down the number of HTTP requests needed to request AWS credentials.

Related issues:

  1. #222 (closed)
  2. #185 (closed)
  3. #210 (closed)

Testing notes

  1. Created an AWS instance with the following IAM instance profile:
{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "VisualEditor0",
            "Effect": "Allow",
            "Action": [
                "s3:PutObject",
                "s3:GetObject",
                "s3:AbortMultipartUpload",
                "s3:DeleteObject"
            ],
            "Resource": "arn:aws:s3:::stanhu-s3-workhorse-testing/*"
        }
    ]
}
  1. On LFS, artifacts, packages, and uploads, I configured the object store connections in the following way:
gitlab_rails['uploads_object_store_connection'] = {
   'provider' => 'AWS',
   'region' => 'us-west-2',
   'use_iam_profile' => true
}
gitlab_rails['uploads_object_store_enabled'] = true
gitlab_rails['uploads_object_store_direct_upload'] = true
gitlab_rails['uploads_object_store_proxy_download'] = true
gitlab_rails['uploads_object_store_remote_directory'] = "stanhu-s3-workhorse-testing"
Edited May 29, 2020 by Stan Hu
Assignee
Assign to
Reviewers
Request review from
Time tracking
Source branch: sh-workhorse-direct-s3