CI file variables
Problem to solve
Often, we want to provide a file, not just some data, to a job running in gitlab-ci. As we see in !24722 (diffs, comment 138652817) , approaches take to this at present are to put encoded versions of the data into the variable and have the CI script echo the data into a file before starting.
Sasha, Software Developer, https://design.gitlab.com/research/personas#persona-sasha
Devon, DevOps Engineer, https://design.gitlab.com/research/personas#persona-devon
Sidney, Systems Administrator, https://design.gitlab.com/research/personas#persona-sidney
Passing large or binary data to a gitlab-runner job is currently frustrating and needs awkward workarounds that have to be identical on both source and destination. We can - and should - do better.
The runner already has support for writing a variable to a file. We hard-code it in certain cases, e.g. for the kubernetes
KUBECONFIG data: https://gitlab.com/gitlab-org/gitlab-ce/blob/master/app/models/project_services/kubernetes_service.rb#L118
file: true in the variable specification, the runner will write the raw value to a temporary file, and populate the variable itself with the path to that file.
We can expose this functionality in the CI/CD variables project settings page, and allow people to specify an arbitrary file to upload instead of converting the data with
xxd or some other hack. Since we'll be storing its content in the database in the first iteration, we should have a low size limit - maybe 4KiB or so, but certainly < 1MiB.
We also need to start tracking whether the variable is a "file variable" in the DB or not. When it is, we just set
file: true in the spec we pass to the runner, and the rest happens by magic
What does success look like, and how can we measure that?
As a developer, when trying to build a new pipeline that relies on the presence of a file in the CI environment (for, e.g., credentiasl), I can simply upload that file in the "CI/CD settings" page and have it available when my job runs with no faffing.
Links / references