Proposal: unified blob storage

Everyone can contribute. Help move this issue forward while earning points, leveling up and collecting rewards.

This is a proposal for a unified solution for blob storage in the gitlab-rails codebase.

Blobs are stored in a universal namespace with string keys. The keys are decoupled from the underlying storage paths. Legacy blobs are mounted in the universal namespace so that they can be accessed transparently. There is no need to backfill or migrate data.

Background

This is meant as an alternative to #348959 (closed). The goals were:

  1. Avoid data migrations, both for SQL records that refer to existing objects, and in object storage itself.
  2. Where possible, preserve existing code and mechanisms. This proposal implies we add a layer underneath the existing system.
  3. Should work equally well when backed by disk storage as when backed by cloud object storage (e.g. S3)

All blobs in the current system are identified by a (location, path) pair, e.g. (:uploads, path/to/my-screenshot.png). These pairs can be 1-1 mapped into a unified namespace of string keys, e.g. uploads/path/to/my-screenshot.png.

Features

  1. Deleting blobs is fast (a SQL update)
  2. Renaming blobs is fast (a SQL update)
  3. No data migrations or backfills required

SQL schema for table blobs

Field Type Comment
id bigserial unique row ID, immutable
key text LIMIT 1024 unique string key, mutable
created_at timestamp creation timestamp
state integer enum: PENDING, READY, DELETED

New storage configuration

  • Default: local storage in /var/opt/gitlab/gitlab-rails/shared/@blob
  • Alternatively, if a (new) object storage configuration unified_blobs is present, store blobs there.
  • Blob paths are derived from row ID's. The mapping is invertible.
    • Blob 1: 00/01/1
    • Blob 1234 (hex: 0x4d2): 04/d2/4d2

Operations

Create

  1. Caller issues Create("foo")
  2. We create SQL row (123, "foo", now(), PENDING)
  3. Workhorse uploads data to blob URL for ID 123 (URL depends on backing store, may be a file:// URL)
  4. When request reaches Rails, update state PENDING to READY

Open

  1. Caller issues Open("artifacts/baz")
  2. If there is a row in table blobs with key "artifacts/baz" and state DELETED, return not found
  3. Create list of potential locations:
  4. New blob with key "artifacts/baz" and state READY
  5. Blob "baz" in legacy artifacts storage location
  6. Try to open each location, return first match

Delete

  1. Caller issues Delete("artifacts/baz")
  2. Find or create "artifacts/baz" in table blobs
  3. Set state of row to DELETED

Note that deleting a blob is a pure SQL transaction. We only mark the blob as deleted so it can no longer be retrieved. Actual deletion happens asynchronously.

Async cleanup

  1. Select batch of rows with state DELETED or (state PENDING and created_at "too long ago")
  2. For each row,
    • Look up legacy storage path
    • Delete blob from legacy storage path
    • Delete blob from new storage path
    • Delete row

Workhorse integration

The new system aims to make minimal changes to Workhorse<->Rails interactions. For direct upload to HTTP object storage backends, nothing changes. For direct upload to file storage backends (the default Omnibus configuration), we must add support for file:// URLs to Workhorse.

To get rid of disk buffered upload, we need to add an internal API route where Workhorse can request a direct upload URL from Rails.

Edited by 🤖 GitLab Bot 🤖