Skip to content

Object Storage: remove direct_upload and background_upload configurations

Quang-Minh Nguyen requested to merge qmnguyen0711/1689-object-storage-remove-direct_upload-and-background_upload-configurations into master

What does this MR do and why?

For gitlab-com/gl-infra/scalability#1689 (closed)

This MR is to remove Object Storage background_upload and direct_upload configurations. In detail, it removes the ability of users to enable background_upload for all types. All object storage types fallback to use direct_upload by default. Some types, such as pages, terraform_state, and secure_files (CI), are not affected as they don't use either background uploading or direct uploading. The corresponding configurations in Omnibus, and CNG will be cleaned up in next MRs.

All code paths are not removed (yet). Instead, we introduce one hidden variable so that the customers could bring back legacy background uploading if something goes wrong when upgrading. After some minor releases, we can get rid of that variable, and clean up the code base completely (following up in gitlab-com/gl-infra/scalability#1690 (closed)).

How to set up and validate locally

  • Case 1: test the default setting. Set the following configuration in config/gitlab.yml 
  lfs:
    enabled: true
    object_store:
      enabled: true
      remote_directory: lfs-objects
      connection:
        provider: AWS
        aws_access_key_id: minio
        aws_secret_access_key: gdk-minio
        region: gdk
        endpoint: 'http://127.0.0.1:9000'
        path_style: true

Run the following code snippet in console:

[1] pry(main)> LfsObjectUploader.direct_upload_enabled?
=> true
[2] pry(main)> LfsObjectUploader.background_upload_enabled?
=> false
  • Case 2: set the same configuration to case 1 in config/gitlab.yml. Start the console with GITLAB_LEGACY_BACKGROUND_UPLOADS=lfs variable
[2] pry(main)> LfsObjectUploader.direct_upload_enabled?
=> false
[3] pry(main)> LfsObjectUploader.background_upload_enabled?
=> true
  • Case 3: The input configurations are ignored. Set the following configuration in config/gitlab.yml 
  lfs:
    enabled: true
    object_store:
      enabled: true
      remote_directory: lfs-objects
      background_upload: true
      direct_upload: false
      connection:
        provider: AWS
        aws_access_key_id: minio
        aws_secret_access_key: gdk-minio
        region: gdk
        endpoint: 'http://127.0.0.1:9000'
        path_style: true
[1] pry(main)> LfsObjectUploader.direct_upload_enabled?
=> true
[2] pry(main)> LfsObjectUploader.background_upload_enabled?
=> false

MR acceptance checklist

This checklist encourages us to confirm any changes have been analyzed to reduce risks in quality, performance, reliability, security, and maintainability.

Merge request reports