Inherit environment variables from dependent jobs
### Problem to solve
CI/CD often needs to pass information from one job to another and artifacts can be used for this, although it's a heavy solution with unintended side effects. Workspaces is another proposal for passing files between jobs. But sometimes you don't want to pass files at all, just a small bit of data.
### Further details
Use cases needing ability to pass information between jobs:
* you create a semantic version during build and want to pass it the deploy job
* you build a docker image and want to pass the ref to another job
* you have an error message that you want to pass to your Slack integration
### Proposal
Considering that https://gitlab.com/gitlab-org/gitlab/issues/17066 provided a workflow to pass custom variables to Rails via a `dotenv` artifact, in this MVC issue we want to allow passing data to other jobs.
We can use `needs` or `dependency` keywords by specifying dependencies of the jobs, which works on both DAG and Stage pipelines.
```yaml
build:
stage: build
script:
- BUILD_VERSION=$(./build) # In script, you get a variable needs to be passed to the other jobs.
- echo "BUILD_VERSION=$BUILD_VERSION" >> build.env # Write the variable in dotenv file.
artifacts:
reports:
dotenv: build.env # Report back dotenv file to rails.
deploy:
stage: deploy
script:
- echo "Build version is #{BUILD_VERSION}" # Consume the variable
dependencies:
- build
# Or, if you'd prepare to use DAG,
#
# needs:
# - build
#
# Or, don't declare anything. All of the jobs in the previous stages are recognized as dependent automatically.
```
### Concerns
Outstanding:
- [ ] Clarify how it affects variable expansion in multiple places e.g. Expansion on Rails.
Addressed:
* On cross-project pipeline or parent-child pipelines, this feature works in the same way that "artifact" works.
* If two variable keys are conflicted from different dotenv files, the system may select based on completion time of build; or by random selection.
* If two jobs in a row update the dotenv file, or if two jobs provide two different ones, then each job has its dotenv file.
### Technical proposal
Here is pseudocode.
```diff
diff --git a/app/models/concerns/ci/contextable.rb b/app/models/concerns/ci/contextable.rb
index 5ff537a7837..2c100ace5b8 100644
--- a/app/models/concerns/ci/contextable.rb
+++ b/app/models/concerns/ci/contextable.rb
@@ -18,6 +18,7 @@ module Ci
variables.concat(deployment_variables(environment: environment))
variables.concat(yaml_variables)
variables.concat(user_variables)
+ variables.concat(depended_variables)
variables.concat(secret_group_variables)
variables.concat(secret_project_variables(environment: environment))
variables.concat(trigger_request.user_variables) if trigger_request
@@ -34,6 +35,16 @@ module Ci
scoped_variables.to_hash
end
+ ##
+ # Inherit variables from dependencies, which specified via `job:needs` or `job:dependencies`
+ # in .gitlab-ci.yml.
+ # TODO: Batch-load
+ def depended_variables
+ ([self] + dependencies).map do |dependency|
+ Ci::JobArtifacts::List::Dotenv.new(dependency).load
+ end
+ end
+
##
# Variables that do not depend on the environment name.
#
```
### Links / references
* https://buddy.works/blog/new-feature-passing-parameters
### Availability and Testing
<!-- What risks does this change pose? How might it affect the quality/performance of the product?
What additional test coverage or changes to tests will be needed?
Will it require cross-browser testing?
See the test engineering process for further guidelines: https://about.gitlab.com/handbook/engineering/quality/test-engineering/ -->
<!-- If cross-browser testing is not required, please remove the relevant item, or mark it as not needed: [-] -->
- [-] [Review and add/update tests for this feature/bug](https://docs.gitlab.com/ee/development/testing_guide/index.html). Consider [all test levels](https://docs.gitlab.com/ee/development/testing_guide/testing_levels.html). See the [Test Planning Process](https://about.gitlab.com/handbook/engineering/quality/test-engineering).
* Add new unit test for job with `dependencies` keyword
* Add new unite test for job with `needs` keyword
- No new integration test or E2E test required
- No `package-and-qa` required
issue