Skip to content

[Discuss] Automated tests for gitlab-openbao CNG images

Problem to solve

The CNG project builds the gitlab-openbao images (Debian, UBI, and FIPS), and pushes these images to registry.gitlab.com/gitlab-org/build/cng/gitlab-openbao. For instance, at the moment the UBI image for OpenBao v2.4.1 are pushed to registry.gitlab.com/gitlab-org/build/cng/gitlab-openbao:2.4.1-ubi.

The gitlab-openbao images are referenced by the Runway service repository and the OpenBao Helm Chart. We have to make sure that they're properly tested before being used in these projects.

Questions

  • How are the gitlab-openbao CNG images tested?
  • Are they tested automatically?
  • Is there anything preventing a broken CNG image from being used in the Runway service or in the Helm Chart?

Answers

Based on the issue comments, here are the answers to the questions in the issue description:

How are the gitlab-openbao CNG images tested?

The gitlab-openbao CNG images are tested through multiple layers:

  1. OpenBao Chart integration tests: The OpenBao Helm Chart repository runs basic e2e tests that deploy the chart to a short-lived cluster and ensure Pods come up healthy. These tests deploy all variants (Debian, UBI, and FIPS) using a parallel:matrix configuration.

  2. GitLab Chart pipelines: When triggered via trigger-chart-test from CNG MR pipelines, the GitLab Chart runs:

    • Unit tests that template the chart and compare rendered manifests
    • Deployment tests (like trigger-eks133) that deploy GitLab Chart into a cluster
    • Minimal API-based specs in spec/features
    • QA tests using gitlab-qa with smoke tests and optionally full suite tests

  3. Renovate-triggered testing: When OpenBao version bumps occur, Renovate opens MRs in the OpenBao chart, which triggers e2e tests deploying all image variants (Debian, UBI, FIPS) to ensure they unseal as expected.

Are they tested automatically?

Partially automated, with gaps:

  • The trigger-chart-test job in CNG is manual and only available for MR and feature branch pipelines, not default branch pipelines
  • Scheduled daily CNG pipelines that rebuild images don't automatically trigger tests
  • Solution implemented: Clemens Beck added a scheduled pipeline in the OpenBao Chart running at 8:00 UTC to test the latest images daily

Is there anything preventing a broken CNG image from being used in the Runway service or in the Helm Chart?

Currently, there are gaps in protection:

  • Images built by scheduled CNG pipelines aren't automatically tested before being available for consumption
  • The Runway service builds on top of CNG images and deploys to staging/production without automated testing of the base CNG image
  • Mitigation: The newly added scheduled pipeline in OpenBao Chart (running at 8:00 UTC after CNG nightly builds at 6:00 UTC) will help catch issues, though it's post-deployment rather than preventative

The discussion also revealed that E2E testing for GitLab Secrets Manager integration is being implemented in issue #573410, which will provide higher-level integration testing once OpenBao is enabled in review environments.

Based on the issue comments, here are the action items:

Action items

  1. Pass GITLAB_VERSION for Debian images in trigger-chart-test

  2. Add scheduled pipeline for OpenBao Chart

    • Running at 8:00 UTC, to test gitlab-openbao images daily after CNG nightly builds
    • To be configured by @clemensbeck
Edited by Fabien Catteau