Skip to content

Add versioning and spec:inputs to AST CI/CD templates

Problem to solve

TLDR; The AST groups must adapt to the new deprecation guidelines that favors migration over breaking changes.

Our primary enablement solution is via CI/CD templates, which integrate our CI based AST scanners into the customer's CI/CD configuration. It is well known to us that the flexibility and the breadth of possibilities offered by our CI/CD capabilities is making it extremely difficult for engineering teams to make changes without risking to impact existing CI/CD configurations.

Though, as of recently, making breaking changes should be avoided at all costs (to simplify). Even the existing 3 months minimun deprecation notice is not deemed acceptable by our biggest customers, who might be required to produce a significant effort to migrate and prevent breaking changes to impact them. As a result, we find ourselves stuck in a situation where we can't move forward without risking to break things and thus not following the new company policy.

Unlike a robust integration system, our current enablement solution for AST scanners using CI/CD template lacks too major traits:

  1. a versioning system to allow changes and improvements to be made without breaking existing usages. This also gives to customers the control over when to migrate, and at what pace.
  2. a clear contract to allow consumers to understand what they can or can't do.

CI/CD components to the rescue, but the cavalry is late

The long term vision is to replace our vendored CI/CD templates with CI/CD components which are actually solving these problems but they unfortunately also bring new ones:

Solving these problems will require time and prioritization beyond AST groups control. It also highly depends on the product and UX vision for CI/CD components in self-managed environments, which is not defined yet. While we can advocate and contribute to these changes, this can't be done entirely by us and without impacting our own roadmaps, which are critical to achieve the company's FY26 vision.

The status quo

In the mean time, we still need to make changes to our CI/CD templates, constantly facing the risk of breaking things and requiring to re-adjust our plans. And this actually keeps happening, like with these recent examples:

  • An S2 incident was caused when adding a new job definition to the Dependency-Scanning template for the new DS analyzer: gitlab-com/gl-infra/production#19310 (closed)
  • Removing old SAST jobs still led to breaking user pipelines and had to be reverted: !179767 (merged)
  • Removing license check rules in the DAST latest template unexpectedly led to no longer runing the job and has to be reverted, forcing the team to look for an alternative and delay the change: #364371

This status quo leads to customer insatisfaction, degrades the perception of reliability and stability of our AST offering, and impacts the efficiency and morale of the engineering teams.

So we need an alternative with the product capabilities that are available today.

Proposal

Here comes "CI/CD templates 2.0".

We actually can already make some enhancements to the CI/CD templates to improve the current situation. This could be interesting for both us and our users to do these, and make a step toward the transition to CI/CD components, until they are ready for our needs.

1. Versioning

While we can't offer a version managememt system as clean as the one available for CI/CD components, based on git tags and releases, we can still do something basic that suits are main needs: using a version suffix in the template filename.

Similarly to what we actually did by introducing the latest template edition, we can introduce a vx suffix: SAST.v2.gitlab-ci.yaml.

This allows us to provide breaking changes without impacting existing customers, by adding them to a new version of the template. This way, rather than pushing the breaking change to the customers and force them to migrate on our terms, they can control when they do the migration.

This basic versioning in the filename also allows to build migration tools to assist users in assesssing the migration impact, cost, and then help them to actually do the migration. For instance, such a tool, coupled with an API token access could create MRs in the relevant projects to bump the template filename in the CI/CD configuration. This would then allow project maintainers to assess the compatibility of the changes before applying them. NB: additional migration tooling might be necessary for enablement done via Policies.

We already have metrics in place for CI/CD templates, tracking new "versions" (aka new files) is thus a straighforward change. This also allows to fit in the new company policy and easily track usage of deprecated templates before we remove them or sunset a related feature.

2. Input specification

The spec:inputs capability is not reserved to CI/CD components and can be used in other include types, like the vendored CI/CD templates we use.

This allows us to define a clear contract and explicit what parameters users can use, while ensuring backward compatibility as we make changes to a given template. We could also maybe limit customization support to these parameters and warn users that other overrides are prone to unannounced and unsupported breaking changes. This brings the expected stability for customers who include a given CI/CD template file and expect things to stay functional until they decide to include a different file with an updated contract.

This prevents users to override the CI job definitions to customize some parameters, which creates a strong dependency between their CI/CD configuration and our template. This is typically what happened for the incident of the Dependency Scanning template update. By using spec:inputs instead, developers now exactly what users could customized and adjust their implementation accordingly.

An other great outcome of using spec:inputs instead of the CI/CD variables we use today is to prevent undesired cross-template side-effects. Indeed, most of our templates include global CI/CD variables which can lead to odd behavior when including several templates and wanting to customize some settings. We often use prefixes to prevent this (e.g. DS_EXCLUDED_PATHS, SAST_EXCLUDED_PATHS). The spec:inputs are local to a given include so there is no side-effect and we can simplify naming. NB: most of our analyzer use a CLI module that automatically grabs input either from the command line options or from environment variables, so we might still have some side-effect from variables defined in the global scope. Though, that's a problem that is also present when using CI/CD components.

Pros

  1. The versioning gives the very much desired and expected stability to our customers.
  2. The versioning allows us to make breaking changes without impacting immediately existing customers but instead offering them control over when to apply them.
  3. The versioning allows us to build migration tools to assist customers in upgrading (a "fully automated migration" might probably be unachievable in some circumstances though).
  4. The versioning allows us to have clear and distinct usage metric per template version.
  5. The versioning prepares users to what's coming to them when we will switch to CI/CD components as our main enablement solution.
  6. The spec:inputs allows us to clearly document the supported customizations
  7. The spec:inputs allows us to prevent undesired cross template side-effects (global CI/CD variables).

Cons

  1. For new template files to be usable with Scan Execution Policies, additional development is required. Though, there is already an option to select a template edition (stable vs latest) so it might be a small efort to make the versions showing up there. There might be some other usability impact to consider and discuss with groupsecurity policies.
  2. Using spec:inputs in a CI/CD template requires to adjust the implementation of the Scan Execution Policy feature. Currently, it can only customize the content of the template configuration it includes by using CI/CD variables. A quick workaround could be to makes these new template use CI/CD variables as fallback for spec:inputs options. Another option could be to defer the usage of spec:inputs to a later iteration, after Scan Execution Policies are ready for such usage.
  3. Proliferation of simultaneously used CI/CD templates. This will create additional maintenance burden and possibly extra investigation step in support cases. This probably will require a thorough tracking of existing versions and maybe a compatibility matrix (though, the vendoring aspects reduces the need for such matrix). That said, this is aligned with the new company policy and additional engineering cost to support migration over breaking changes has been accounted for in the decision to apply such policy.
  4. This proposal could be seen as an unecessary iteration toward CI/CD components. This arguments mainly depends on how fast progress is made on CI/CD components and how urgent is our need for new and improved temlates. The former is unknown, the later is known and it's today. Additionally, once user migrate to use CI/CD components, they have to adopt versioning management and configuration via spec:inputs. So the migration from old temlates to these new enhanced templates should be pretty close to migrating to CI/CD components directly.

Example use cases

To support this proposal, here are a couple of use cases that would directly and immediately benefit from it.

Migrate from Gemnasium to the new DS analyzer

Due to the no longer acceptable breaking changes, we had to revisit our rollout strategy. And then again due to the incident. This currently leaves us in a situation where new users enabling the Dependency Scanning feature will onboard with the deprecated Gemnasium analyzer by default. To use the new DS analyzer, users will be required to enable it explicitely via an additional configuration change.

With the suggested versioning in place, groupcomposition analysis can release a Dependency-Scanning.v2.gitlab-ci.yaml file, which removes all the old Gemnasium jobs, and only uses the new DS analyzer job by default. This new template will be the one we document as our main enablement solution for our [Dependency Scanning feature] (https://docs.gitlab.com/user/application_security/dependency_scanning/dependency_scanning_sbom/#configuration) and in the migration guide

Existing customers can stay with the old template using the deprecated/no longer supported Gemansium analyzer until they decide to migrate. We can track usage of the old template and incentivize users to migrate to the new version that uses the new DS analyzer. Finally, when usage is below a given threshold, we can decide to sunset the old Gemnasium analyzer and remove the old template.

Integrate Static Reachability into SCA offering

As we want to bring Static Reachability to Beta this quarter, and to GA later, we have to revisit its integration (currently in SAST.latest.gitlab-ci.yml) and brings it closer to our SCA features. Though, this involves making significant changes to ourt templates, which again creates a risk of breaking existing customer's pipelines.

The safest approach for us is to keep using the latest edition of the template which we advertise as possibly unstable and prone to unannounced breaking changes. Though this presents unfortunate downsides:

  • users of the latest templates are not always aware of the risk of having their pipeline broken
  • adoption of Static Reachability will be drastically reduced, impairing our ability to receive feedback

Migrate from old SAST analyzers to GLAS

Like the transition from Gemnasium to the new DS analyzer, it might be beneficial to follow a similar pattern for the consolidation of SAST analyzers around GitLab Advanced SAST.

Support MR pipelines in AST templates

Due to the same risks of breaking changes, we've delayed the support of MR pipelines in our stable templates for nearly 2 years now. We only brought that support to the latest templates, creating the anti-pattern of forcing customers who wanted that feature to use the unstable templates.

We finally decided to move on this year and allow customers to benefit from them in the stable templates, but only as an opt-in feature, again to prevent breaking existing CI/CD configuration.

With versionned templates and spec:inputs, we could give all our users a much better experience. For intance, we could release a new version of the templates, that enable the MR pipelines feature by default, which better aligns with our own guidelines and saves resources to our customers. Then we could use a clear input parameter for users who want to disable them and use branch pipeline instead.

Using MR pipelines by default:

include:
  - template: `SAST.v2.gitlab-ci.yml`

Disabling MR pipelines:

include:
  - template: `SAST.v2.gitlab-ci.yml`
    inputs:
      enable_mr_pipelines: false

Implementation plan

  • consider all existing stable (unversioned) CI/CD templates as the v1.
    • use an implicit approach by adding a comment to the current stable template file
    • or take an explicit approach and add a x.v1.gitlab-ci.yml file for all the existing stable CI/CD templates. Copy the current content, and then include this v1 file in the original templates. For example, add a SAST.v1.gitlab-ci.yml file with the content from the SAST.gitlab-ci.yml file, then modify the SAST.gitlab-ci.yml file content to be: 
      include:
  - template: "Jobs/SAST.v1.gitlab-ci.yml"
  • add a x.v2.gitlab-ci.yml file as needed, e.g. when a breaking change is required to be made on a given stable template. Keep bumping the version as new risky changes are made to the template.
  • adjust Scan Execution Policies logic to allow to select a particular version for a given template, on top of the stable vs latest edition.
  • adjust Scan Execution Policies to allow to set inputs parameters when including a template
  • develop migration tooling to assess impacted projects and assist in migrating them to a new version of a given template.
Edited by Olivier Gonzalez