Skip to content

Support Workhorse directly uploading files to S3 [3/3]

Stan Hu requested to merge sh-workhorse-direct-s3 into master

This adds the AWS client directly to Workhorse and a new configuration section for specifying credentials. This makes it possible to use S3 buckets with KMS encryption and proper MD5 checksums.

This is disabled by default. For this to be used:

  1. GitLab Rails needs to send the UseWorkhorseClient and RemoteTempObjectID in the /authorize endpoint. (gitlab!29389 (merged))

  2. S3 configuration must be specified in config.toml, or Rails must be configured to use IAM instance profiles (use_iam_profile in Fog connection parameters).

S3 sessions are created lazily and cached for 10 minutes to avoid unnecessary local I/O access. When IAM instance profiles are used, this also cuts down the number of HTTP requests needed to request AWS credentials.

Related issues:

  1. #222 (closed)
  2. #185 (closed)
  3. #210 (closed)

Testing notes

  1. Created an AWS instance with the following IAM instance profile:
{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "VisualEditor0",
            "Effect": "Allow",
            "Action": [
                "s3:PutObject",
                "s3:GetObject",
                "s3:AbortMultipartUpload",
                "s3:DeleteObject"
            ],
            "Resource": "arn:aws:s3:::stanhu-s3-workhorse-testing/*"
        }
    ]
}
  1. On LFS, artifacts, packages, and uploads, I configured the object store connections in the following way:
gitlab_rails['uploads_object_store_connection'] = {
   'provider' => 'AWS',
   'region' => 'us-west-2',
   'use_iam_profile' => true
}
gitlab_rails['uploads_object_store_enabled'] = true
gitlab_rails['uploads_object_store_direct_upload'] = true
gitlab_rails['uploads_object_store_proxy_download'] = true
gitlab_rails['uploads_object_store_remote_directory'] = "stanhu-s3-workhorse-testing"
Edited by Stan Hu

Merge request reports