[Discuss] Automated tests for gitlab-openbao CNG images

Problem to solve

The CNG project builds the gitlab-openbao images (Debian, UBI, and FIPS), and pushes these images to registry.gitlab.com/gitlab-org/build/cng/gitlab-openbao. For instance, at the moment the UBI image for OpenBao v2.4.1 are pushed to registry.gitlab.com/gitlab-org/build/cng/gitlab-openbao:2.4.1-ubi.

The gitlab-openbao images are referenced by the Runway service repository and the OpenBao Helm Chart. We have to make sure that they're properly tested before being used in these projects.

Questions

  • How are the gitlab-openbao CNG images tested?
  • Are they tested automatically?
  • Is there anything preventing a broken CNG image from being used in the Runway service or in the Helm Chart?

Based on the information that GitLab Chart doesn't have integration tests for the OpenBao Chart (as explained in #573410), here are the updated sections:

Answers

How are the gitlab-openbao CNG images tested?

The gitlab-openbao CNG images are tested through multiple layers:

  1. OpenBao Chart integration tests: The OpenBao Helm Chart repository runs basic e2e tests that deploy the chart to a short-lived cluster and ensure Pods come up healthy. These tests deploy all variants (Debian, UBI, and FIPS) using a parallel:matrix configuration.

  2. GitLab Chart pipelines: When triggered via trigger-chart-test from CNG MR pipelines, the GitLab Chart runs:

    • Unit tests that template the chart and compare rendered manifests
    • However, GitLab Chart does NOT currently test the OpenBao Chart integration
    • Deployment tests (like trigger-eks133) deploy GitLab but without OpenBao
    • QA tests using gitlab-qa with smoke tests don't include Secret Manager functionality

  3. Renovate-triggered testing: When OpenBao version bumps occur, Renovate opens MRs in the OpenBao chart, which triggers e2e tests deploying all image variants (Debian, UBI, FIPS) to ensure they unseal as expected.

  4. GitLab rails pipelines: Rails pipelines use the GitLab chart to deploy a test environment for e2e tests. These currently do not deploy OpenBao or execute any Secret Manager related e2e tests.

Are they tested automatically?

Limited automation with significant gaps:

  • The trigger-chart-test job in CNG is manual and only available for MR and feature branch pipelines, not default branch pipelines
  • Scheduled daily CNG pipelines that rebuild images don't automatically trigger tests
  • Most critically: GitLab Chart pipelines don't test OpenBao integration at all

Is there anything preventing a broken CNG image from being used in the Runway service or in the Helm Chart?

There are significant gaps in protection:

  • Images built by scheduled CNG pipelines aren't automatically tested before being available for consumption
  • The Runway service builds on top of CNG images and deploys to staging/production without automated testing of the base CNG image

Mitigation: The newly added scheduled pipeline in OpenBao Chart (running at 8:35 UTC after CNG nightly builds at 6:00 UTC) provides some coverage, though it's post-deployment rather than preventative

Action Items

  1. ✅ Pass GITLAB_VERSION for Debian images in trigger-chart-test

  2. ✅ Add scheduled pipeline for OpenBao Chart

  3. Enable OpenBao integration testing in GitLab Chart

    • Critical gap: GitLab Chart currently doesn't test OpenBao Chart integration
    • Requires updating default test values in autodevops.sh to configure OpenBao
    • Basic features specs that deploy the OpenBao Chart and validate Secret Manager functionality. This could be implemented by enabling the feature in a project using the GraphQL API.
    • Additionally, we might update CNG dev docs and provide guidance on when to trigger trigger-chart-test to test OpenBao integration, or trigger it automatically when the gitlab-openbao directory changes.
    • TODO: Open an issue for this.

Note: QA/E2E tests on CNG are covered by #573410.

Edited by 🤖 GitLab Bot 🤖