Support a sandbox test harness runner for Terraform projects

Related to gitlab-com/content-sites/handbook!19292

Related to gitlab-org/orbit/knowledge-graph#444 (closed)

Support a sandbox test harness runner for Terraform projects

Summary

Add a CI component to common-ci-tasks that enables integration testing of Component Ownership Modules against Hackystack sandbox accounts for AWS and GCP, using Terraform's native test framework (terraform test).

Motivation

Component Ownership Module authors currently have no standardized way to run integration tests against real cloud infrastructure in CI. Providing a reusable CI component that handles authentication, provisioning, test execution, and teardown against sandbox accounts would significantly improve module quality and reduce the risk of deploying untested infrastructure changes.

Proposal

How it works

  1. The module author includes the CI component in their .gitlab-ci.yml and configures it with the target sandbox account identifier (AWS Account Number or GCP Project ID).
  2. On push, the CI pipeline:
    • Authenticates to the sandbox account using OIDC.
    • Runs terraform test, which provisions short-lived resources defined in .tftest.hcl files and executes assertions against them.
    • Tears down all resources on completion.
  3. No Atlantis integration is required. Provisioning and teardown are handled directly by CI/CD jobs for simplicity.

Authentication

Authentication to sandbox accounts is handled via GitLab OIDC. The OIDC trust relationship must be bootstrapped in the target sandbox account using the appropriate module:

Resource isolation via prefix

Module authors should be encouraged to accept an optional prefix variable (defaulting to an empty string) on all resource names. This allows multiple concurrent test runs to target the same sandbox account without resource naming collisions. The CI component should generate a unique prefix per pipeline (e.g. derived from $CI_PIPELINE_ID) and pass it through to the test configuration.

Teardown and cleanup

The primary teardown mechanism is terraform test itself, which destroys all resources it creates at the end of a test run. As a safety net for leaked resources, the component should optionally support running a cleanup tool such as aws-nuke or cloud-nuke against the sandbox account on a scheduled basis or as a post-test step.

Implementation considerations

  • The CI component should expose configuration for:
    • Cloud provider (aws or gcp)
    • Sandbox account identifier (AWS Account Number / GCP Project ID)
    • Terraform version (GL_ASDF_TERRAFORM_VERSION)
    • Optional: cleanup tool toggle (aws-nuke / cloud-nuke)
  • Terraform test files (.tftest.hcl) live alongside the module source in the consuming project. The CI component does not need to generate these.
  • Consider supporting command = plan-only (unit test) runs as a lighter-weight default, with full command = apply (integration test) runs gated on a manual trigger or specific branches.
  • Document the OIDC bootstrapping process clearly, including the minimum IAM permissions / roles required in the sandbox account for the test harness to function.

Open questions

  • Should the CI component enforce a maximum TTL or cost ceiling on sandbox resources to prevent runaway spend?
    • Running a regular cleanup may help here.
  • What is the desired behavior when terraform test teardown partially fails? Retry, alert, or defer to scheduled nuke?
    • Probably a combination of all three: several retries, a failed pipeline, plus a deferred nuke.
  • Should we provide a shared Hackystack sandbox account per-team, or expect module authors to bring their own?
    • Module authors to bring their own

cc @reprazent @fforster @stejacks-gitlab @pguinoiseau

Edited by Andrew Newdigate