Skip to content

Support consolidated object store configuration in Charts

Gerard Hickey requested to merge 2158-consolidated-storage into master

Summary

Add support of the consolidated object storage within charts via global.appConfig.object_store. Have workhorse consume this to configure it's own object_storage(.s3) settings as well.

Implementation

Adds global.appConfig.object_store configuration to the charts. This allows the the configuration of a single connection property, which will be consumed by Rails (Sidekiq, Webservice, Task-runner) to configure lfs, uploads, artifacts, ( ... ). This does not include backups, as consumed by task-runner. That is not integrated with the Rails/Rake codebase, and as such does not currently have the capability.

This is implemented in such a fashion that the included MinIO (when used) will be propagated in this way. When not using an external object storage provider, this will result in a single /etc/gitlab/objectstorage/object_store connection file, which Rails itself will consume & populate into all the individual configurations.

When using external storage, such as GCS, Amazon S3, or Azure blob storage (coming soon) this will also be the case.

Noticable difference

Configuring object storage can now be much simpler than as previously, where the settings need to be used with YAML anchors or copy-pasted when the connection configuration / secrets where identical

  object_store:
    enabled: true
    proxy_download: true # default
    connection:
      secret: my-aws-secret
      key: connection
  artifacts:
    enabled: true            # default, on/off of feature
    bucket: gitlab-artifacts # default
  uploads:
    enabled: true            # default, on/off of feature
    bucket: gitlab-uploads   # default
  lfs:
    enabled: true            # default, on/off of feature
    bucket: gitlab-lfs       # default
provider: AWS
aws_access_key_id: AWS_ACCESS_KEY_ID
aws_secret_access_key: AWS_SECRET_ACCESS_KEY
region: us-east-1

When using the provider: AWS, upload handling can be handled entirely by gitlab-workhorse, without reliance on pre-signed URLs from gitlab-webservice. This brings direct support for Content-MD5 header, which should address problems with ETag mismatch and bring greater support for AWS S3 bucket encryption.

Note: Only S3 compatible object storage providers are supported in combination with workhorse offloading currently.

Closes #2158 (closed)

Edited by Jason Plum

Merge request reports