Skip to content

Draft: Adding Secure Files Uploader / Downloader

Darby Frey requested to merge secure_files_model into master

UPDATE: These changes have been split up and moved to the MRs below:

  1. Adding Secure Files data model and file uploader
  2. Adding Secure Files API

What does this MR do and why?

This change adds the data model, uploader, and APIs to support the Secure Files mechanism discussed in #346290 (closed).

The feature is designed to support folks who need a simple way to add binary files to their CI pipelines. The use case I'm focused on is in the code signing process for mobile teams, but this approach has been generalized to support other use cases as well.

This first iteration supports basic API interactions. Future iterations will support direct integration with the GitLab Runner, as well as a UI for file uploads and viewing (discussion here).

I recorded a demo of this feature as well: https://youtu.be/eK3FUskHfdo

File Attributes

Secure Files have several attributes which are stored in the database:

  • project Secure Files belong to a project
  • name the name of the file will be downloaded as
  • file the file itself, files are encrypted before they are written to disk
  • checksum a checksum of the unencrypted file is computed and saved, this is checked by the download API
  • permissions an enum of read_only, read_write, or execute, to be used by the future GitLab Runner integration

APIs

Upload a Secure File

Secure Files can be uploaded via POST /projects/:id/secure_files with the name, file (5 MB limit) and optional permissions inputs. For example:

curl --request POST --header "PRIVATE-TOKEN: <your_access_token>" https://gitlab.example.com/api/v4/projects/1/secure_files --form "name=myfile.jks" --form "file=@/path/to/file/myfile.jks"

The response payload includes the file id, name, checksum, checksum_algorithm, and permissions.

{
  "id": 1,
  "name": "myfile.jks",
  "checksum": "16630b189ab34b2e3504f4758e1054d2e478deda510b2b08cc0ef38d12e80aac",
  "checksum_algorithm": "sha256",
  "permissions": "read_only"
}

List all Secure Files for a Project

A list of Secure Files for a project is returned via GET /projects/:id/secure_files. The response payload is similar to the above.

curl --request GET --header "PRIVATE-TOKEN: <your_access_token>" https://gitlab.example.com/api/v4/projects/1/secure_files"

[{
  "id": 1,
  "name": "myfile.jks",
  "checksum": "16630b189ab34b2e3504f4758e1054d2e478deda510b2b08cc0ef38d12e80aac",
  "checksum_algorithm": "sha256",
  "permissions": "read_only"
}]

Get an individual Secure File

An individual Secure File is returned via GET /projects/:id/secure_files/:secure_file_id with a similar response payload.

curl --request GET --header "PRIVATE-TOKEN: <your_access_token>" https://gitlab.example.com/api/v4/projects/1/secure_files/1"

{
  "id": 1,
  "name": "myfile.jks",
  "checksum": "16630b189ab34b2e3504f4758e1054d2e478deda510b2b08cc0ef38d12e80aac",
  "checksum_algorithm": "sha256",
  "permissions": "read_only"
}

Download a Secure File

A single Secure File can be downloaded via GET /projects/:id/secure_files/:secure_file_id/download. There is no JSON response for this endpoint, it simply returns the decrypted file content.

curl --request GET --header "PRIVATE-TOKEN: <your_access_token>" https://gitlab.example.com/api/v4/projects/1/secure_files/1/download --output myfile.jks

Optionally, the checksum returned from the GET endpoint can be used to ensure the correctness of the downloaded file.

Delete an individual Secure File

An individual Secure File can be deleted via DELETE /projects/:id/secure_files/:secure_file_id

curl --request DELETE --header "PRIVATE-TOKEN: <your_access_token>" https://gitlab.example.com/api/v4/projects/1/secure_files/1"

To Do

There are still a few things I need to work out with this change, but I wanted to share it early for feedback. I'll keep the list below updated as I work through the remaining items and feedback.

  • I don't fully understand how Geo Replicators work, so I've excluded this uploader for the time being. I would want to determine if this is needed before moving forward. UPDATE: Thanks to @mkozono for helping with this. I've created a follow up issue to add Geo support for this feature later #349893 (closed)
  • A feature flag rollout issue is still needed. UPDATE: based on the feature flag handbook, I don't think a feature flag is necessary for this feature https://about.gitlab.com/handbook/product-development-flow/feature-flag-lifecycle/#when-to-use-feature-flags
  • I believe a change in Workhorse is needed in order to support the added file uploader, I will be looking for additional guidance on those changes. UPDATE: Workhorse acceleration won't be utilized for this feature for the time being.
  • I don't fully understand the impact of including the new model in the import_export/all_models.yml so that should be better understood before moving forward. UPDATE: This point has been resolved for now. Secure Files won't be included in project exports.

How to set up and validate locally

  1. The feature flag must be enabled in order for the APIs to work. This can be done with:

    Feature.enable(:ci_secure_files)

  2. A local personal access token will be needed to execute any of the API requests

  3. With the feature flag enabled, the API examples above can be used to validate this change.

MR acceptance checklist

This checklist encourages us to confirm any changes have been analyzed to reduce risks in quality, performance, reliability, security, and maintainability.

Edited by Darby Frey

Merge request reports