Skip to content

Added Import::Offline::Configuration model and client

What does this MR do and why?

This creates Import::Offline::Configuration with a dew database table, a new client for connecting to object storage locations, and implements a check to determine if the object storage bucket is accessible before creating the offline export.

Notable implementation decisions:
  • Instead of updating BulkImports::Configuration, I opted to create a new model since object storage configuration attributes have no overlap with bulk import configuration attributes.
  • Given security concerns with MinIO (confidential comment), MinIO is currently only a valid provider when in development environments. In the future, MinIO will be enabled by an application setting that's disabled by default: #579705

References and object storage notes

  • Resolves Update BulkImports::Configuration to support of... (#537510) and Create object storage client for offline transfer (#537512) as the client was implemented here
  • Fog::Storage uses fog-aws to connect to AWS S3 buckets and MinIO buckets because MinIO also implements the S3 API.
  • path_style: true means that Fog will attempt to connect to an object storage location by including the bucket name in the path, e.g. https://s3.my-minio.com/bucket-name/object-key. path_style: false refers to virtual hosted style where the bucket name is part of the host name, e.g. https://bucket-name.s3.amazonaws.com/object-key. MinIO installations use path style by default but can be configured to use virtual hosted style. Newer AWS buckets must be accessed using virtual hosted style, but older buckets may still use path style.

How to set up and validate locally

This MR does not implement any real functionality yet, so local testing should be done in the console. RSpec tests should pass, but they don't make any actual connection to an object storage provider. To test locally:

Set up an object storage bucket

Using MinIO configured with GDK (if not already done):

  1. Enable object storage locally: https://gitlab-org.gitlab.io/gitlab-development-kit/howto/object_storage/
  2. Set the region on your GDK MinIO instance. In the MinIO UI (accessible at http://127.0.0.1:9000/ by default), go to Configuration > Region and set the region (gdk is used in this example)

Using an AWS bucket (more involved than MinIO, but closer to what users would use):

  1. Follow the instructions to provision an AWS sandbox account: https://handbook.gitlab.com/handbook/company/infrastructure-standards/realms/sandbox/#how-to-get-started
  2. Create an access key by navigating to IAM service > Users > your user > Security credentials > Access keys > Create access key
  3. Create an object storage bucket by navigating to S3 service and creating a new bucket. Keep note of the bucket name and the region it's hosted in to use later.

To test:

  1. Run database migrations
  2. Enable :offline_transfer_exports feature flag
  3. Open a rails console and create an Import::Offline::Export using Import::Offline::Exports::CreateService:
current_user = User.first # whatever user you want
source_hostname = 'https://offline-environment-gitlab.example.com' # Does not need to match the actual instance address
portable_params = [
  { type: 'group', full_path: 'gnuwget' },
  { type: 'project', full_path: 'toolbox/gitlab-smoke-tests' }
]
provider = :minio
bucket = 'import-objects' # Any existing bucket works and no files will be created yet
credentials = {
  aws_access_key_id: 'minio',
  aws_secret_access_key: 'gdk-minio',
  region: 'gdk',
  endpoint: 'http://127.0.0.1:9000', # No endpoint for AWS buckets
  path_style: true # Set to false for AWS buckets
}

result = Import::Offline::Exports::CreateService.new(current_user, source_hostname, portable_params, provider, bucket, credentials).execute

# result should be success and payload should be the new Import::Offline::Export.

To see test the created Import::Offline::Configuration directly:

configuration = Import::Offline::Export.last.configuration
client = Import::Clients::ObjectStorage.new(
  fog_provider: configuration.fog_provider,
  bucket: configuration.bucket,
  credentials: configuration.object_storage_credentials
)
client.validate_configuration! # Returns nil if connection is successful

MR acceptance checklist

Evaluate this MR against the MR acceptance checklist. It helps you analyze changes to reduce risks in quality, performance, reliability, security, and maintainability.

Related to #537510

Edited by Sam Word

Merge request reports

Loading