## Background The Step Runner currently uses various contexts (namespaces) in its expression language to access different types of variables. As the feature evolves and more variable sources are introduced, the current naming scheme creates potential conflicts and lacks clarity about variable precedence and sources. This epic proposes a comprehensive restructuring of these contexts to provide better organization, prevent naming conflicts, and enable future extensibility. New contexts should be implemented on-demand based on validated customer needs. ## Definitions - `Expression language`: the language used to access a variable in steps, components, or a pipeline template. Expressions have the form `$[[ ... ]]` or `${{ ... }}` - `Context`: A namespace used in an expression to group variables. For example, the `input` context is used in the expression `$[[ input.my_var ]]` to access `my_var` ## As is To access: - A job variable in a job definition: `$MY_VAR` - An input in a component: `$[[ inputs.my_var ]]` - An input in a pipeline template: `$[[ inputs.my_var ]]` - A job input in a job definition: `${{ job.inputs.my_var }}` - A job output in a job definition: `${{ jobs.my_job.outputs.my_var }}` (likely, not yet decided) - A step input in a step: `${{ inputs.my_var }}` - A step output in a step: `${{ steps.my_step.outputs.my_var }}` - An environment variable in a step: `${{ env.MY_VAR }}` - A job variable in a step: `${{ job.my_var }}` ## Problems The current as-is creates the following issues: - There is a namespace clash between `${{ job.my_var }}` and `${{ job.inputs.my_var }}` if the user creates a job variable named `inputs` - Users should be able to access a job variable from a known source (project var, predefined var, etc) so that they can be sure it has not been overwritten later. An extensible approach should allow for this - For example, GitLab predefined variables should be immutable, yet can be overridden by variables defined in the job definition - Changes to how variables are accessed is a breaking change, so these changes should be decided and implemented as soon as is practical - The current `job` context is ambiguous - it's unclear whether it refers to job-specific variables, job metadata, or job configuration - No clear way to distinguish between different variable sources, making debugging and troubleshooting difficult - Limited extensibility for future variable sources (runner-specific, instance-level, etc.) ## Proposal The contexts used in the expression languages should change to the following: | Before | After | Notes | |--------------------------------------------|--------------------------------------|-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| | `inputs.<var-name>` (in component) | (no change) | Accessed using `$[[ ]]`, does not conflict with steps | | `inputs.<var-name>` (in pipeline template) | (no change) | Accessed using `$[[ ]]`, does not conflict with steps | | `job.inputs.<var-name>` | (no change) | Likely the most obvious way to access job inputs | | `jobs.<job-name>.outputs.<var-name>` | (no change) | Likely the most obvious way to access job outputs | | `inputs.<var-name>` (in step) | (no change) | Accessing step inputs in a step does not change, it will not be used at the same time as component or pipeline template inputs. This changes if steps can be defined inline in a job | | `steps.<step-name>.outputs.<var-name>` | (no change) | `steps` does not change, apart from being renamed to `functions` when the Steps is renamed to CI Functions | | `env.<var-name>` | (no change) | `env` context does not change | | `job.<var-name>` | `vars.<var-name>` | `job` context renamed to `vars`, groups all job variables | | (new) | `job.vars.<var-name>` | Defined in individual jobs in .gitlab-ci.yml ~~or when manually triggered~~ (manually triggered variables are discouraged and may be deprecated) | | (idea abandoned) | ~~`pipeline.vars.<var-name>`~~ | ~~Variables passed to downstream pipelines, trigger variables (API calls), scheduled pipeline variables, manual pipeline run variables, variables from pipeline API creation and manual job variables.~~ Abandoned because these are being discouraged and may be deprecated | | (new) | `project.vars.<var-name>` | Defined in project Settings > CI/CD | | (new) | `group.<group-name>.vars.<var-name>` | Defined in group Settings > CI/CD | | (new) | `runner.vars.<var-name>` | Defined by a runner admin in the runner config | | (new) | `instance.vars.<var-name>` | Defined by an admin on the GitLab instance | | (new) | `gitlab.vars.<var-name>` | Predefined variables in GitLab | | (new) | `jobs.<job-name>.result.<var-name>` | Job results could be automatically captured in a `result` sub-context, for example `jobs.my_job.result.status`, `jobs.my_job.result.duration` | | (new) | `steps.<step-name>.result.<var-name>` | Step results could be automatically captured in a `result` sub-context, for example `steps.my_step.result.status`, `steps.my_step.result.duration` | | (new) | `mr.<var-name>` | There may be additional useful contexts depending on the situation, such as `mr` for job run in a merge request pipeline | ## Benefits This proposal provides several key advantages: - **Clear variable source identification**: Users can explicitly reference variables from specific sources, preventing confusion about variable precedence - **Namespace conflict resolution**: Eliminates potential conflicts between job variables and other contexts - **Enhanced debugging**: Clear context names make it easier to identify where variables originate - **Future extensibility**: The hierarchical structure allows for easy addition of new variable sources - **Immutable access**: Users can access GitLab predefined variables directly without worrying about overrides - **Consistent naming**: All variable contexts follow the pattern `source.vars.variable_name` ## Implementation concerns - Initial implementation should focus on completing breaking changes, i.e. job variables should be fetched from `vars.XXX`, not `job.XXX` - Issues can be created for each new context (`pipeline`, `group`, `runner`, `gitlab`, etc). These should not be played until a customer requests the functionality - The source of a job variable needs to be sent to the Runner from GitLab Rails - Documentation updates required across all Step Runner materials - Consider providing a deprecation period with warnings before fully removing old context support ## Success Criteria - New context system provides clear variable source identification - No namespace conflicts exist in the new system - Performance impact is minimal - Documentation is comprehensive and clear - Migration path is well-defined and communicated ## References Related to https://gitlab.com/gitlab-com/content-sites/handbook/-/merge_requests/14199+