Skip to content

Implement Dependency proxy via Workhorse injectors

Igor Drozdov requested to merge id-dependency-proxy-workhorse-inject-2 into master

The Dependency Proxy allows users to reduce risk and reliance on external dependencies by caching frequently used packages for fast, reliable access. However, since the Rails app downloads the blobs from upstream sources, it's possible that while downloading is in progress, it will block other requests.

In order to fix the issue, we plan to move the implementation to Workhorse.

    Client->>Workhorse: GET /v2/*group_id/dependency_proxy/containers/*image/blobs/:sha
    Workhorse->>Rails: GET /v2/*group_id/dependency_proxy/containers/*image/blobs/:sha 
    Rails->>Rails: Check DB. Is blob persisted in cache?
    alt In Cache
        Rails->>Workhorse: Respond with send-url injector
        Workhorse->>Client: Send the file to the client
    else Not In Cache
        Rails->>Rails: Generate auth token and download URL for the blob in upstream registry
        Rails->>Workhorse: Respond with send-dependency injector
        Workhorse->>Container Registry: Open stream for a blob
        Container Registry->>Workhorse: Stream
        Workhorse->>Rails: GET /v2/*group_id/dependency_proxy/containers/*image/blobs/:sha/authorize
        Rails->>Workhorse: Respond with upload instructions
        Workhorse->>Client: Send the file to the client
        Workhorse->>Object Storage: Save the file
        Workhorse->>Rails: Finalize the upload

Each of the plain approaches (injectors or pre-auth) has a flaw:

The idea is to combine both approaches:

  • When blob is requested:
    • If it exists in the database, it's sent via Workhorse (send_upload)
    • If it doesn't exist, Workhorse downloads it from the remote and streams the file to the user; after that, it uploads the file via Workhorse using a pre-authorized request

Related issue: #335563 (closed)

Edited by Igor Drozdov

Merge request reports