Skip to content

Allow incomplete bucket for Secure Files in object storage config

Stan Hu requested to merge sh-allow-secure-files-bucket-optional into master

What does this MR do and why?

This restores the changes made in !131819 (merged) that was reverted in !149479 (merged). If ci_secure_files were configured in a storage-specific settings, this would previously disable consolidated storage settings.

This change can now go in now that !149480 (merged) has been merged. That merge request relaxes the constraints when consolidated settings can be used so that consolidated settings will still function if ci_secure_files is set in storage-specific settings with the same connection.

This commit applies the suggestion in #414673 (comment 1546146600) to add the ci_secure_files config to SUPPORTED_TYPES and ALLOWED_INCOMPLETE_TYPES so that the consolidated storage config can be used, but not produce an error if the bucket configuration is missing.

This change resolves the issues that caused the original change to be reverted Revert adding ci_secure_files to consolidated o... (!122905 - merged) and allows us to add support for consolidated object storage without introducing a breaking change.

Changelog: fixed

MR acceptance checklist

Please evaluate this MR against the MR acceptance checklist. It helps you analyze changes to reduce risks in quality, performance, reliability, security, and maintainability.

Screenshots or screen recordings

Screenshots are required for UI changes, and strongly recommended for all other merge requests.

Before After

How to set up and validate locally

  1. Set up your GDK with consolidated storage. For example:
object_store:
  connection:
    provider: AzureRM
    azure_storage_account_name: YOUR-ACCOUNT-NAME
    azure_storage_access_key: YOUR-ACCESS-KEY
  consolidated_form: true
  enabled: true
  objects:
    artifacts:
      bucket: test1/artifacts
    external_diffs:
      bucket: test1/external_diffs
    lfs:
      bucket: test1/lfs
    uploads:
      bucket: test1/uploads
    packages:
      bucket: test1/packages
    dependency_proxy:
      bucket: test1/dependency-proxy
    terraform_state:
      bucket: test1/terraform
    pages:
      bucket: test1/pages
    ci_secure_files:
      bucket: test1/ci_secure_files
    gitaly_backups:
      bucket: test1/gitaly_backups
  1. In gitlab/config/gitlab.yml configure a storage-specific settings:
  ## CI Secure Files
  ci_secure_files:
    enabled: true
    object_store:
      enabled: true
      connection:
        <copy the connection settings from `gitlab.object_store`> 
  1. Run bin/rails console.
  2. Check that Settings.ci_secure_files.object_store['consolidated_settings'] is true.
  3. Check that Settings.ci_secure_files.object_store has the right settings.
  4. Check that Settings.artifacts.object_store has the right settings too.
Edited by Stan Hu

Merge request reports