Consolidate object storage config in one place
Currently each object type has its own section in gitlab.yml. For example, for artifacts:
artifacts: path: tmp/tests/artifacts enabled: true object_store: enabled: false remote_directory: artifacts # The bucket name background_upload: false connection: provider: AWS aws_access_key_id: AWS_ACCESS_KEY_ID aws_secret_access_key: AWS_SECRET_ACCESS_KEY region: us-east-1
External diffs, LFS, uploads, packages, etc. all have similar independent configuration object storage sections. While this redundancy makes it possible to configure each bucket with different providers or credentials, this causes a configuration explosion that makes GitLab hard to manage.
This change preserves the legacy format but adds a new
section that enforces a single, common object storage provider for all
object storage types. This will make it possible for the S3 client in
Workhorse to operate with one credential and simplify the configuration
for the end user. An example config:
object_store: enabled: true connection: provider: AWS aws_access_key_id: AWS_ACCESS_KEY_ID aws_secret_access_key: AWS_SECRET_ACCESS_KEY region: us-east-1 proxy_download: true objects: artifacts: bucket: artifacts proxy_download: false external_diffs: bucket: external-diffs lfs: bucket: lfs-objects uploads: bucket: uploads packages: bucket: packages dependency_proxy: bucket: dependency_proxy
- The consolidated config only gets used if
object_storeis NOT defined within the types themselves.
- A bucket needs to be defined for each object type.
proxy_downloadcan be overridden from the common configuration.
Consolidating support for a single bucket for all types is a larger and more involved change.
First step of #23345