Skip to content

Enforce template inclusion in pipelines

Problem to Solve

As a customer with compliance requirements, I may want to enforce that a particular job (security scan, auditing task, etc.) runs on every pipeline.

Instance vs. Group Level

There are two obvious initial places where we could implement this feature: at the group or the instance level. What we're hearing from our large enterprise customers who want to implement this feature is that they want a centralized group with broad-based (cross-group) authority to implement these pipelines and enforce their inclusion. They want to avoid any possibility of a non-administrator user creating a group or project that can "evade" the instance-level requirements. Most of the urgent demand we're hearing for this feature is also coming from this group. For these reasons, we've chosen to provide the instance-level configuration first, via this issue.

That being said, many other users at the group level will also be interested in this feature for compliance reasons, as well as for consistency and code reuse of templates within their group, and we want to empower those teams with this feature as well. In order to achieve that goal as well, we will follow up this MVC with https://gitlab.com/gitlab-org/gitlab-ee/issues/11343, which will bring the feature to the group level and make it available to everyone - including gitlab.com users. This next iteration will also give more control to enterprises as well, since they will be able to implement different policies for different groups underneath their centrally controlled process requirements.

Intended users

The intended users for this iteration are GitLab system administrators. However, they will use this to fulfill the requirements dictated on them by Director and above level individuals who are responsible for the audibility of code deployment processes at the organization.

Governance policies set at the organizational level require assurance that certain items are complete before deploying code to various environments. This feature allows for a "gate" that is required to be run on every pipeline before it is considered a "success."

Further details

In many ways, we see this as the machine-readable extension of merge request approval rules. Those rules allow you to specify very specifically which groups or individuals must be humans-in-the-loop for a merge process (the typical gate towards deployment of code).

In the same way, this feature will be the machine-in-the-loop analog to merge request approvers, allowing for automated processes to be required before a merge request is merged.

Some financial customers have built a custom version of this by:

  • Create a DevOps repository with yml files with the required job definition(s)
  • Instance wide project template with yml file with the DevOps files referenced in include:
  • Post-commit hooks to a custom app to enforce no one removing the include: statement

Proposal

At the instance level, allow an administrator to create a required include: that runs in every pipeline created on the instance.

Allow instance administrators to create a job (a portion of a .gitlab-ci.yml) to be included automatically every time GitLab calculates the jobs for a pipeline.

Leverage existing instance template repository functionality to allow the user to select one of the .yml files to be included in every pipeline. Thus the user journey to enable this is:

  • Create a repository (or go to the existing instance template repository)
  • Create a gitlab-ci-required.yml or similar
  • Add to that file all of the required jobs, stages, etc.
  • Ensure that repository is selected as the template repository here: image
  • Select the file as designed here:
  • When GitLab creates any pipeline on that instance, it will automatically include that file and its jobs into the pipeline as if the file was referenced in an include:

Solution

NOTE: Further design discussion will happen inside of the merge request https://gitlab.com/gitlab-org/gitlab-ee/merge_requests/13923. That concept aligns with UX

What does success look like, and how can we measure that?

One consideration should be: how can we make sure the user cannot override the file with "yaml squashing."

Permissions and Security

This feature will only be available to instance administrators using the existing pages that are only available to them.

Links / references

This feature also relates to https://about.gitlab.com/solutions/financial-services-regulatory-compliance/.

Customer's Interested

Edited by Jason Yavorska