Consolidate object storage config in one place

Merged Stan Hu requested to merge sh-consolidate-object-storage-config into master

Currently each object type has its own section in gitlab.yml. For example, for artifacts:

  artifacts:
    path: tmp/tests/artifacts
    enabled: true
    object_store:
      enabled: false
      remote_directory: artifacts # The bucket name
      background_upload: false
      connection:
        provider: AWS
        aws_access_key_id: AWS_ACCESS_KEY_ID
        aws_secret_access_key: AWS_SECRET_ACCESS_KEY
        region: us-east-1

External diffs, LFS, uploads, packages, etc. all have similar independent configuration object storage sections. While this redundancy makes it possible to configure each bucket with different providers or credentials, this causes a configuration explosion that makes GitLab hard to manage.

This change preserves the legacy format but adds a new gitlab.yml section that enforces a single, common object storage provider for all object storage types. This will make it possible for the S3 client in Workhorse to operate with one credential and simplify the configuration for the end user. An example config:

  object_store:
    enabled: true
    connection:
      provider: AWS
      aws_access_key_id: AWS_ACCESS_KEY_ID
      aws_secret_access_key: AWS_SECRET_ACCESS_KEY
      region: us-east-1
    proxy_download: true
    objects:
      artifacts:
        bucket: artifacts
        proxy_download: false
      external_diffs:
        bucket: external-diffs
      lfs:
        bucket: lfs-objects
      uploads:
        bucket: uploads
      packages:
        bucket: packages
      dependency_proxy:
        bucket: dependency_proxy

Note that:

  1. The consolidated config only gets used if object_store is NOT defined within the types themselves.
  2. A bucket needs to be defined for each object type.
  3. Only bucket, enabled, and proxy_download can be overridden from the common configuration.

Consolidating support for a single bucket for all types is a larger and more involved change.

First step of #23345 (closed)

Edited by Stan Hu