Next

Problem to solve

CI/CD often needs to pass information from one job to another and artifacts can be used for this, although it's a heavy solution with unintended side effects. Workspaces is another proposal for passing files between jobs. But sometimes you don't want to pass files at all, just a small bit of data.

Further details

Use cases needing ability to pass information between jobs:

• you create a semantic version during build and want to pass it the deploy job
• you build a docker image and want to pass the ref to another job
• you have an error message that you want to pass to your Slack integration

Proposal

Considering that #17066 (closed) provided a workflow to pass custom variables to Rails via a dotenv artifact, in this MVC issue we want to allow passing data to other jobs.

We can use needs or dependency keywords by specifying dependencies of the jobs, which works on both DAG and Stage pipelines.

build:
  stage: build
  script:
    - BUILD_VERSION=$(./build)                            # In script, you get a variable needs to be passed to the other jobs.
    - echo "BUILD_VERSION=$BUILD_VERSION" >> build.env    # Write the variable in dotenv file.
  artifacts:
    reports:
      dotenv: build.env                                   # Report back dotenv file to rails.

deploy:
  stage: deploy
  script:
    - echo "Build version is #{BUILD_VERSION}"            # Consume the variable 
  dependencies:
    - build
  # Or, if you'd prepare to use DAG,
  #
  # needs:
  #  - build
  #
  # Or, don't declare anything. All of the jobs in the previous stages are recognized as dependent automatically.

Concerns

Outstanding:

• Clarify how it affects variable expansion in multiple places e.g. Expansion on Rails.

Addressed:

• On cross-project pipeline or parent-child pipelines, this feature works in the same way that "artifact" works.
• If two variable keys are conflicted from different dotenv files, the system may select based on completion time of build; or by random selection.
• If two jobs in a row update the dotenv file, or if two jobs provide two different ones, then each job has its dotenv file.

Technical proposal

Here is pseudocode.

diff --git a/app/models/concerns/ci/contextable.rb b/app/models/concerns/ci/contextable.rb
index 5ff537a7837..2c100ace5b8 100644
--- a/app/models/concerns/ci/contextable.rb
+++ b/app/models/concerns/ci/contextable.rb
@@ -18,6 +18,7 @@ module Ci
         variables.concat(deployment_variables(environment: environment))
         variables.concat(yaml_variables)
         variables.concat(user_variables)
+        variables.concat(depended_variables)
         variables.concat(secret_group_variables)
         variables.concat(secret_project_variables(environment: environment))
         variables.concat(trigger_request.user_variables) if trigger_request
@@ -34,6 +35,16 @@ module Ci
       scoped_variables.to_hash
     end
 
+    ##
+    # Inherit variables from dependencies, which specified via `job:needs` or `job:dependencies`
+    # in .gitlab-ci.yml.
+    # TODO: Batch-load
+    def depended_variables
+      ([self] + dependencies).map do |dependency|
+        Ci::JobArtifacts::List::Dotenv.new(dependency).load
+      end
+    end
+
     ##
     # Variables that do not depend on the environment name.
     #

Links / references

• https://buddy.works/blog/new-feature-passing-parameters

Availability and Testing

• [-] Review and add/update tests for this feature/bug. Consider all test levels. See the Test Planning Process.
  • Add new unit test for job with dependencies keyword
  • Add new unite test for job with needs keyword
• No new integration test or E2E test required
• No package-and-qa required