Proposal for adding binary file support to CI variables
In order to be able to add Code Signing capabilities to GitLab, we will need to add the ability to support binary files as CI variables. I'm opening up this issue to share a proposed implementation and gather feedback. Some parts of this had been discussed in the issue above, but I wanted to scope the discussion for this proposal to the first iteration.
Problem
The problem today is that file type CI variables don't support binary files (only text files). For teams building mobile apps, binary files are needed as configuration for the code signing process.
The workaround that folks use today is to base64 encode their files and then add them as environment variables. Then, as part of their CI script, they will base64 decode the variables to write the files in the runner.
We can better support these folks by adding the ability to upload those binary files directly to GitLab.
Proposal
Data Model
The proposed change is to create a new SecureFile model. This model would store the metadata about a given file, and the actual file would be stored encrypted in object storage. The model would contain the following attributes:
- project_id (belongs to project)
- file name
- file reference (object storage / disk location)
- checksum (checksum of the unencrypted file used for file verification)
- permissions (
read_only,read_write, orexecutefile mode when persisting the file in Runner) - timestamps
Upload API
For the first iteration, binary files would only be added via an API - a UI for this feature would come later. (UI discussion here).
A new API will be created to support a request like this:
curl --request POST --header "PRIVATE-TOKEN: <your_access_token>"
"https://gitlab.example.com/api/v4/projects/1/secure_files"
--form "filename=upload_keystore.jks"
--form "file=@/path/to/binary/file"A file size limit should be enforced for this API as well, starting with a 5 MB limit.
Get API
The get API would return the metadata for an individual files (or all files for the project).
curl --request GET --header "PRIVATE-TOKEN: <your_access_token>" https://gitlab.example.com/api/v4/projects/1/secure_files/1"This request returns the file id, name, checksum and permissions. For example:
{
"id": 1,
"name": "myfile.jks",
"checksum": "16630b189ab34b2e3504f4758e1054d2e478deda510b2b08cc0ef38d12e80aac",
"checksum_algorithm": "sha256",
"permissions": "read_only"
}
Download API
The download API would be the API the Runner would use to download the file. The request would look like:
curl --request GET --header "PRIVATE-TOKEN: <your_access_token>"
"https://gitlab.example.com/api/v4/projects/1/secure_files/1/download" This endpoint would simply return the decrypted file contents for the given file id.
Runner
The Runner would need to be updated to pull in Secure Files as part of the job setup. This change will be addressed in a later issue. For now, there is a simple ruby script that can be used to enable this functionality in a CI job.