Skip to content
GitLab Next
  • Menu
Projects Groups Snippets
  • Help
    • Help
    • Support
    • Community forum
    • Submit feedback
    • Contribute to GitLab
  • Sign in / Register
  • GitLab GitLab
  • Project information
    • Project information
    • Activity
    • Labels
    • Members
  • Repository
    • Repository
    • Files
    • Commits
    • Branches
    • Tags
    • Contributors
    • Graph
    • Compare
    • Locked Files
  • Issues 43,148
    • Issues 43,148
    • List
    • Boards
    • Service Desk
    • Milestones
    • Iterations
    • Requirements
  • Merge requests 1,404
    • Merge requests 1,404
  • CI/CD
    • CI/CD
    • Pipelines
    • Jobs
    • Schedules
    • Test Cases
  • Deployments
    • Deployments
    • Environments
    • Releases
  • Packages & Registries
    • Packages & Registries
    • Package Registry
    • Container Registry
    • Infrastructure Registry
  • Monitor
    • Monitor
    • Metrics
    • Incidents
  • Analytics
    • Analytics
    • Value stream
    • CI/CD
    • Code review
    • Insights
    • Issue
    • Repository
  • Snippets
    • Snippets
  • Activity
  • Graph
  • Create a new issue
  • Jobs
  • Commits
  • Issue Boards
Collapse sidebar
  • GitLab.org
  • GitLabGitLab
  • Issues
  • #22638
Closed
Open
Created Jun 07, 2018 by Mark Pundsack@markpundsackContributor0 of 1 task completed0/1 task

Inherit environment variables from dependent jobs

Problem to solve

CI/CD often needs to pass information from one job to another and artifacts can be used for this, although it's a heavy solution with unintended side effects. Workspaces is another proposal for passing files between jobs. But sometimes you don't want to pass files at all, just a small bit of data.

Further details

Use cases needing ability to pass information between jobs:

  • you create a semantic version during build and want to pass it the deploy job
  • you build a docker image and want to pass the ref to another job
  • you have an error message that you want to pass to your Slack integration

Proposal

Considering that #17066 (closed) provided a workflow to pass custom variables to Rails via a dotenv artifact, in this MVC issue we want to allow passing data to other jobs.

We can use needs or dependency keywords by specifying dependencies of the jobs, which works on both DAG and Stage pipelines.

build:
  stage: build
  script:
    - BUILD_VERSION=$(./build)                            # In script, you get a variable needs to be passed to the other jobs.
    - echo "BUILD_VERSION=$BUILD_VERSION" >> build.env    # Write the variable in dotenv file.
  artifacts:
    reports:
      dotenv: build.env                                   # Report back dotenv file to rails.

deploy:
  stage: deploy
  script:
    - echo "Build version is #{BUILD_VERSION}"            # Consume the variable 
  dependencies:
    - build
  # Or, if you'd prepare to use DAG,
  #
  # needs:
  #  - build
  #
  # Or, don't declare anything. All of the jobs in the previous stages are recognized as dependent automatically.

Concerns

Outstanding:

  • Clarify how it affects variable expansion in multiple places e.g. Expansion on Rails.

Addressed:

  • On cross-project pipeline or parent-child pipelines, this feature works in the same way that "artifact" works.
  • If two variable keys are conflicted from different dotenv files, the system may select based on completion time of build; or by random selection.
  • If two jobs in a row update the dotenv file, or if two jobs provide two different ones, then each job has its dotenv file.

Technical proposal

Here is pseudocode.

diff --git a/app/models/concerns/ci/contextable.rb b/app/models/concerns/ci/contextable.rb
index 5ff537a7837..2c100ace5b8 100644
--- a/app/models/concerns/ci/contextable.rb
+++ b/app/models/concerns/ci/contextable.rb
@@ -18,6 +18,7 @@ module Ci
         variables.concat(deployment_variables(environment: environment))
         variables.concat(yaml_variables)
         variables.concat(user_variables)
+        variables.concat(depended_variables)
         variables.concat(secret_group_variables)
         variables.concat(secret_project_variables(environment: environment))
         variables.concat(trigger_request.user_variables) if trigger_request
@@ -34,6 +35,16 @@ module Ci
       scoped_variables.to_hash
     end
 
+    ##
+    # Inherit variables from dependencies, which specified via `job:needs` or `job:dependencies`
+    # in .gitlab-ci.yml.
+    # TODO: Batch-load
+    def depended_variables
+      ([self] + dependencies).map do |dependency|
+        Ci::JobArtifacts::List::Dotenv.new(dependency).load
+      end
+    end
+
     ##
     # Variables that do not depend on the environment name.
     #

Links / references

  • https://buddy.works/blog/new-feature-passing-parameters

Availability and Testing

  • [-] Review and add/update tests for this feature/bug. Consider all test levels. See the Test Planning Process.
    • Add new unit test for job with dependencies keyword
    • Add new unite test for job with needs keyword
  • No new integration test or E2E test required
  • No package-and-qa required
Edited May 20, 2020 by Thao Yeager
Assignee
Assign to
Time tracking