Skip to content

Object Store direct upload

Alessio Caiazza requested to merge object-store-direct-upload into master

LFS Direct Uploader from discussions on gitlab-org/gitlab-ee#4184 (based on !215 (closed))

It needs API changes on GitLab Rails gitlab-org/gitlab-ee!4233 but it's fully backward compatible.

The behavior change is API driven by GitLab rails

Here follow a sketchy sequence diagram

sequenceDiagram
    participant r as user
    participant w as gitlab-workhorse
    participant u as gitlab-unicorn
    participant os as Object Storage

    activate r
    r->>+w: git push with LFS object
    
    w->>+u: authorize
    Note over u,os: Presigning URLs for CarrierWave cache files
    u->>+os: pre-sign PutObject
    os-->>-u: presigned_url
    u->>+os: pre-sign RemoveObject
    os-->>-u: presigned_url
    u-->>-w: presigned_urls

    w->>+os: PutObject
    os-->>-w: result
 
    Note over w,os: Now we hijack the request body with object path and other metadata
    w->>+u: proxy to finalize_upload
    u->>+os: copy cache object to his final localtion
    os-->>-u: 

   u-->>-r: 
   deactivate r
   Note over w,os: now we can delete cache file
   w->>+os: RemoveObject
   os-->>-w: 
   deactivate w

This is the first iteration to achieve https://gitlab.com/gitlab-org/gitlab-ee/issues/4184#note_55980619

Edited by Alessio Caiazza

Merge request reports