Inherit environment variables from dependent jobs
Problem to solve
CI/CD often needs to pass information from one job to another and artifacts can be used for this, although it's a heavy solution with unintended side effects. Workspaces is another proposal for passing files between jobs. But sometimes you don't want to pass files at all, just a small bit of data.
Further details
Use cases needing ability to pass information between jobs:
- you create a semantic version during build and want to pass it the deploy job
- you build a docker image and want to pass the ref to another job
- you have an error message that you want to pass to your Slack integration
Proposal
Considering that #17066 (closed) provided a workflow to pass custom variables to Rails via a dotenv
artifact, in this MVC issue we want to allow passing data to other jobs.
We can use needs
or dependency
keywords by specifying dependencies of the jobs, which works on both DAG and Stage pipelines.
build:
stage: build
script:
- BUILD_VERSION=$(./build) # In script, you get a variable needs to be passed to the other jobs.
- echo "BUILD_VERSION=$BUILD_VERSION" >> build.env # Write the variable in dotenv file.
artifacts:
reports:
dotenv: build.env # Report back dotenv file to rails.
deploy:
stage: deploy
script:
- echo "Build version is #{BUILD_VERSION}" # Consume the variable
dependencies:
- build
# Or, if you'd prepare to use DAG,
#
# needs:
# - build
#
# Or, don't declare anything. All of the jobs in the previous stages are recognized as dependent automatically.
Concerns
Outstanding:
-
Clarify how it affects variable expansion in multiple places e.g. Expansion on Rails.
Addressed:
- On cross-project pipeline or parent-child pipelines, this feature works in the same way that "artifact" works.
- If two variable keys are conflicted from different dotenv files, the system may select based on completion time of build; or by random selection.
- If two jobs in a row update the dotenv file, or if two jobs provide two different ones, then each job has its dotenv file.
Technical proposal
Here is pseudocode.
diff --git a/app/models/concerns/ci/contextable.rb b/app/models/concerns/ci/contextable.rb
index 5ff537a7837..2c100ace5b8 100644
--- a/app/models/concerns/ci/contextable.rb
+++ b/app/models/concerns/ci/contextable.rb
@@ -18,6 +18,7 @@ module Ci
variables.concat(deployment_variables(environment: environment))
variables.concat(yaml_variables)
variables.concat(user_variables)
+ variables.concat(depended_variables)
variables.concat(secret_group_variables)
variables.concat(secret_project_variables(environment: environment))
variables.concat(trigger_request.user_variables) if trigger_request
@@ -34,6 +35,16 @@ module Ci
scoped_variables.to_hash
end
+ ##
+ # Inherit variables from dependencies, which specified via `job:needs` or `job:dependencies`
+ # in .gitlab-ci.yml.
+ # TODO: Batch-load
+ def depended_variables
+ ([self] + dependencies).map do |dependency|
+ Ci::JobArtifacts::List::Dotenv.new(dependency).load
+ end
+ end
+
##
# Variables that do not depend on the environment name.
#
Links / references
Availability and Testing
- [-] Review and add/update tests for this feature/bug. Consider all test levels. See the Test Planning Process.
- Add new unit test for job with
dependencies
keyword - Add new unite test for job with
needs
keyword
- Add new unit test for job with
- No new integration test or E2E test required
- No
package-and-qa
required