CI file variables
Problem to solve
Often, we want to provide a file, not just some data, to a job running in gitlab-ci. As we see in https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/24722/diffs#note_138652817 , approaches take to this at present are to put encoded versions of the data into the variable and have the CI script echo the data into a file before starting.
Target audience
-
Sasha, Software Developer, https://design.gitlab.com/research/personas#persona-sasha
-
Devon, DevOps Engineer, https://design.gitlab.com/research/personas#persona-devon
-
Sidney, Systems Administrator, https://design.gitlab.com/research/personas#persona-sidney
Further details
Passing large or binary data to a gitlab-runner job is currently frustrating and needs awkward workarounds that have to be identical on both source and destination. We can - and should - do better.
Proposal
The runner already has support for writing a variable to a file. We hard-code it in certain cases, e.g. for the kubernetes KUBECONFIG
data: https://gitlab.com/gitlab-org/gitlab-ce/blob/master/app/models/project_services/kubernetes_service.rb#L118
By providing file: true
in the variable specification, the runner will write the raw value to a temporary file, and populate the variable itself with the path to that file.
We can expose this functionality in the CI/CD variables project settings page, and allow people to specify an arbitrary file to upload instead of converting the data with xxd
or some other hack. Since we'll be storing its content in the database in the first iteration, we should have a low size limit - maybe 4KiB or so, but certainly < 1MiB.
We also need to start tracking whether the variable is a "file variable" in the DB or not. When it is, we just set file: true
in the spec we pass to the runner, and the rest happens by magic
What does success look like, and how can we measure that?
As a developer, when trying to build a new pipeline that relies on the presence of a file in the CI environment (for, e.g., credentiasl), I can simply upload that file in the "CI/CD settings" page and have it available when my job runs with no faffing.
Links / references
cc @jlenny