Variable precedence controls in pipeline execution policies (#16430) · Epics · Epics · GitLab.org

Variable precedence controls in pipeline execution policies

# Release notes  Security teams often face a challenging dilemma when attempting to manage and enforce security scans and other requirements centrally. It's critical to ensure security scans are executed when and where they should, but project teams may be required to provide specific inputs to properly execute the analyzers. This results in a delicate balance between security assurance and developer experience. Some examples of balancing enforcement of security scans against developer flexibility include: * Enforcing container scanning policies that couldn't adapt to project-specific container image paths (`CS_IMAGE`). * The inability to allow variables like `SAST_EXCLUDED_PATHS` while blocking risk ones like `SAST_DISABLED`. * Flexibility to define globally shared credentials that are secured (masked/hidden) through the global CI/CD variables, such as `AWS_CREDENTIALS`, and used within Pipeline Execution Policies. With this release, we're providing flexible variable control for Pipeline Execution Policies, giving security teams precise configuration options to manage which variables can be customized by project teams. With the new `variables_override` configuration, security teams can now define their configuration as follows: ```yaml variables_override: allowed: true/false exceptions: [VARIABLE_1, VARIABLE_2] ``` This powerful feature supports two approaches: * **Lock variables** (`allowed: false`): Lock all variables except specific ones listed in exceptions * **Allow variables** (`allowed: true`): Allow all variables to be customized while restricting any critical risks as exceptions **Real-world impact** This enhancement bridges the gap between security requirements and development flexibility: * Security teams can enforce standardized scanning while allowing project-specific customizations * Developers maintain control over project-specific variables without requesting policy exceptions * Organizations can implement consistent security policies without disrupting development workflows * Migration from Compliance Pipelines becomes seamless with even more precise variable control than before By solving this critical variable control challenge, GitLab enables organizations to implement robust security policies without sacrificing the flexibility teams need to deliver software efficiently. # Problem to solve  As customers work to strike a balance between security/compliance and developer velocity, enforcement of CI jobs within project pipelines is only one dimension. There is also a desire to control the behavior for managing and manipulating variables for jobs in a project enforced by a pipeline execution policy. Here are the most common examples: * I want to enforce a container security scan, where all of the rules/configuration are enforceable, but project teams are able to provide the path to a valid container. * I want to enforce security scan, but lock variables (and prevent them from being modified) for variables that may disable the scans or manipulate the results. * I want to ensure policy job variables have the highest precedence, but that downstream project variables otherwise honor the CI precedence. * I want to migrate from compliance pipelines with minimal/no impact to current variable precedence behaviors. # Intended users  1. [Amy, Application Security Engineer](/handbook/product/personas/#amy-application-security-engineer) 2. [Alex, Security Operations Engineer](/handbook/product/personas/#alex-security-operations-engineer) # Proposal 1 - Configure allowed/denied variables in PEP  1. Allow policy creators to define rules in the pipeline execution policy that dictate impact to variable precedence where policies are enforced. 2. By default, allow jobs to be injected gracefully with minimal impact to development team projects - a permissive state where project-level precedence is honored. 3. Allow policy creators to define rules to enforce variables globally, such that all policy variables have highest precedence. 4. Granular enforcement per job, allowing users to lock/define variables per job, allowing users to modify variables for jobs that are exempt from the policy. 5. As this could be challenging to troubleshoot, we'd also need some clear visibility into what is being impacted by policies. I suspect this would need to be written in the job logs at a minimum. #### UX - YAML Mode: ``` # Global definition in policy.yml # Allow everything, but deny specific user-defined variables variables_override: allowed: true exceptions: [SAST_DISABLED, SAST_EXCLUDED_ANALYZERS] # (optionally) We could expose all scanner-disabling variables as a CI variable for an easier use exceptions: $CI_SCANNER_DISABLING_VARIABLES # Or allow user-defined variables to override these variables. Other variables will be ignored. variables_override: allowed: false exceptions: [DAST_PROFILE, DAST_WEBSITE] # Per-job definition (not included at the moment in the implementation plans) secret_detection: variables_override: allowed: false ``` #### UX - Policy Editor: TBD # Proposal 2 - Configure `.inputs.pipeline-execution-policy` template to use in PEP configuration 1. Create an `inputs` template and define allowed inputs based on the CI configuration/rules. ([docs](https://docs.gitlab.com/ci/yaml/inputs/)) 2. Configure a pipeline execution policy that references the include in the yaml configuration along with defined inputs. 3. Provide templates to make it possible to convert from existing CI templates to use of templates with inputs (either supported by the policies team or AST teams). # Solution Analysis - Pros/Cons ### Engineering Assessment ([from PoC MR](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/185834#using-inputs)): > ### Using inputs > > * Templates can be adapted to use `inputs` instead of variables. This will allow for more targeted customization without opening the doors fully by accepting all variables > * Policy makers can customize the templates with static values, or pass ENV variables as inputs to open up the customization to downstream projects > * Advantages: > * Better security by using opt-in approach for customization (lock-down by default) > * We don't have to apply "magical" high precedence for policy jobs to enforce scanners, which would solve the issue with `override_project_ci` and downstream project variables > * Downsides: > * The current templates don't support the `inputs` and have to be redone > * Scanners rely on ENV variables. For example: > * https://gitlab.com/gitlab-org/security-products/analyzers/secrets/-/blob/master/main.go?ref_type=heads#L63 > * https://gitlab.com/gitlab-org/security-products/analyzers/secrets/-/blob/master/analyze.go#L24 > * https://gitlab.com/gitlab-org/security-products/analyzers/container-scanning/-/blob/master/lib/gcs/environment.rb#L9 > * Currently, there's no way to transform `inputs` into ENV variables. The following could solve it? > * https://gitlab.com/groups/gitlab-org/-/epics/16321#note_2387966135 > * https://gitlab.com/gitlab-org/gitlab/-/issues/525110 > * If users prefer opt-out approach (e.g. only disallow means of disabling the scanners via `SECRET_DETECTION_DISABLED`, etc.), all customization variables have to be provided as `inputs` ### Product Assessment: Using inputs: 1. Appears potentially viable in the long-term if we can address UX issues and requirements for managing sensitive data (secrets, tokens, credentials) that need to be passed downstream, restricted from modification, masked/hidden similar to CI variables, and generally securely managed (e.g. with the future secrets manager). 2. Today, UX is a challenge as it requires an additional layer of configuration and maintenance to create a PEP: 1. Create PEP with rules/config stored in the `policy.yml`. 2. Reference the CI configuration that is used by the PEP (the `pipeline-execution-policy.yml`) which in this case would store an `include` with `inputs`. 3. Create the `inputs` template used to define the `spec`. \<--- this would be yet another config file to maintain to achieve the one task. 3. Secrets cannot be used in `inputs` as there is no way to hide/mask the values. There are use cases where PEP jobs need to be configured that make use of tokens to execute the jobs and this must be managed securely, not in plaintext. 4. Maintenance/support of additional templates is an unnecessary burden, whether it's on security policies or AST teams to manage. Even if AST teams agreed, there would be time required to roll out changes and PEP limits/challenges would remain until the templates were available and adopted by customers. Perhaps if AST supported components and components were GA on all distributions, this would be more viable in the long-term. 5. It's not yet clear if/how multiple policies may interact together and the workflow for managing this. For example, two policies using different sets of required inputs for the same analyzer for different sets of projects, or enforced through two different teams. # Risks - We will need to ensure our plans align with that of Scan Execution Policies so the behaviors are not too divergent and cause UX challenges. See https://gitlab.com/groups/gitlab-org/-/epics/11797+. - There may be multiple locations where the default behavior for users is modified which can be impactful, especially if we are not transparent. Adding job logs may be one area we can do this, but there are many views leveraged where more communication could be necessary, such as pipeline views and CI/CD variables settings. # Further details  # Permissions and Security  # Documentation  # Availability & Testing  # Available Tier  # Feature Usage Metrics  # What does success look like, and how can we measure that?  # What is the type of buyer?  # Is this a cross-stage feature?  # What is the competitive advantage or differentiation for this feature? # Links / references  # Next Steps (as of Mar 26, 2025) - [ ] Continue with PoC for variable override controls for PEP. Outline pros/cons and any other outcomes. Grant: I've added a section in the description that we can use to ensure the summarized details/outcomes are captured in one location. - [ ] Grant to revisit an updated timeline to account for slippage. - [ ] Verify + Policies to meet again in 1-2 weeks (Week of Apr 7 at latest) and review progress in the PoC for variable override controls for PEP. Goal: Discuss complexity/challenges vs viability to proceed. Decide there on next steps.  _This page may contain information related to upcoming products, features and functionality. It is important to note that the information presented is for informational purposes only, so please do not rely on the information for purchasing or planning purposes. Just like with all projects, the items mentioned on the page are subject to change or delay, and the development, release, and timing of any products, features, or functionality remain at the sole discretion of GitLab Inc._   > [!important] > > This page may contain information related to upcoming products, features and functionality. It is important to note that the information presented is for informational purposes only, so please do not rely on the information for purchasing or planning purposes. Just like with all projects, the items mentioned on the page are subject to change or delay, and the development, release, and timing of any products, features, or functionality remain at the sole discretion of GitLab Inc.   > [!important] > This page may contain information related to upcoming products, features and functionality. > It is important to note that the information presented is for informational purposes only, so please do not rely on the information for purchasing or planning purposes. > Just like with all projects, the items mentioned on the page are subject to change or delay, and the development, release, and timing of any products, features, or functionality remain at the sole discretion of GitLab Inc.

epic