CI support for triggering a Flux sync

Everyone can contribute. Help move this issue forward while earning points, leveling up and collecting rewards.

• Close this issue

Release notes

Previously, it was not possible to track a pull-based deployment as part of a GitLab CI/CD pipeline. This made it difficult to have pre- and post-deployment jobs and to track deployments across several environments in a single place. The current GitLab release ships with an experimental version to trigger a Flux reconciliation from the pipeline.

We consider this an experiment as we are looking for feedback for the best approach to track the reconciliation of the resources that actually interest our users, not just one resource thoughfully picked by us. If you would like to contribute to this discussion, your insights are welcome in

Problem to solve

As a Platform engineer, I want the pull-based deployments to be part of my deployment pipeline.

If Flux watches a git ref, the sync will happen

• after a new commit arrived to the watched ref
• Flux's regular sync loop fires (default: 10mins)

This does not allow any workflows to run pre- and post-deployment.

In Example deployment project using features curre... (#389857 - closed) we looked into alternative approaches where Flux would not watch the default branch, but either another branch or an OCI artifact generated from the default branch.

Using an OCI artifact is likely the best approach and is inline with where the industry is going.

The idea is to introduce new CI syntax to

• easily build an OCI image from a given directory (like pages jobs deploy public)
• push the built OCI image to the GitLab container registry
• notify Flux (in the selected cluster) about a new image being available
• wait for Flux to sync (or pick up an ongoing sync)
  • we should wait for a Kustomization (health checks, wait) or Helm release to sync (wait in Helm), not just the source
• continue the pipeline

Current workaround

1. trigger a flux sync from the pipeline (see a demo at https://gitlab.com/gitlab-org/ci-cd/deploy-stage/demos/podtato-head-delivery/-/blob/main/.gitlab-ci.yml?ref_type=heads#L63)
2. wait for the flux resource status using kubectl wait (example)

Proposal

The first iteration will consist of a helper script/tool that performs the actions listed above:

• Create an OCI artifact from manifests contained in a directory
• Push the artifact to GitLab's registry
• Wait for Flux to sync the changes (either via notifying Flux explicitly, or waiting for the next cycle)

The tool will be packaged into a public Docker image, which can be used by any CI job that requires the functionality. See https://gitlab.com/gitlab-org/terraform-images for a similar example.

Definition of done

• Script/tool created to package and push OCI artifact.
  • sync waits for the OCI image only
• Image containing script/tool is available under gitlab-org/cluster-integration
• Example job documented

Out of scope (iteration 1)

• Changes to CI syntax

Intended users

• Priyanka (Platform Engineer)

Feature Usage Metrics

• number of times when the job with the given syntax is triggered

This page may contain information related to upcoming products, features and functionality. It is important to note that the information presented is for informational purposes only, so please do not rely on the information for purchasing or planning purposes. Just like with all projects, the items mentioned on the page are subject to change or delay, and the development, release, and timing of any products, features, or functionality remain at the sole discretion of GitLab Inc.

/label workflowrefinement