Add file-based connector for offline and isolated networks

Problem to solve

This issue covers the creation of a network connector to import package metadata for instances that are offline or segregated from the public internet and cannot reach the public bucket with the package metadata (this is an alternative path to Add GCP bucket connector for fetching package m... (#383797 - closed))

Proposal

For GitLab installs that are not connected to the public internet, an alternative to fetching data from a public GCP bucket is needed. Using a file-based import system is proposed where the bucket contents are copied manually (via rake task or by user themselves) and then a rake task is triggered to import the data in the copied file.

In order to facilitate this functionality 2 things are needed:

• a rake task for an admin or installing user to pull bucket data down into a file-based format
• a connector for the sync service (#383722 (closed)) to save this data into the package metadata database

Implementation Plan

When their instance can't reach the bucket, a user will download the archive file and transfer it to a "known" location on their instance: $GITLAB_HOME/vendor/package_metadata_db

• Add PackageMetadata::FileConnector.
  • Implement #fetch_objects_after(sequence_id, chunk_id).
    1. List sequence objects in bucket (these are lexicographically ordered).
    2. Skip any objects less than sequence and chunk until next "unread" object is found. If sequence is not found, start from the first sequence.
    3. Open stream to the object, convert text lines to csv.
    4. Yield converted lines to caller.
• Update sync service (implementation issue: #383722 (closed)) to support the new connector.
• Update the quick start guide for offline instances. Include documented example that includes:
  • How to fetch the license database files using gsutils rsync.
  • How to fetch the license database files using jq and cURL.
  • Where to store the license database files so that they trigger the offline file based connector.

This page may contain information related to upcoming products, features and functionality. It is important to note that the information presented is for informational purposes only, so please do not rely on the information for purchasing or planning purposes. Just like with all projects, the items mentioned on the page are subject to change or delay, and the development, release, and timing of any products, features, or functionality remain at the sole discretion of GitLab Inc.