Implement a dependency proxy like feature for Terraform and OpenTofu providers

Problem to solve

Organizations using Terraform or OpenTofu in their CI/CD pipelines face several challenges:

  1. Network reliability: Every terraform init or tofu init downloads providers from external registries (registry.terraform.io, registry.opentofu.org), creating a dependency on external network availability
  2. Air-gapped environments: Teams operating in secure, network-restricted, or air-gapped environments cannot easily use Terraform/OpenTofu without manually mirroring providers
  3. Pipeline performance: Repeatedly downloading large provider binaries (some exceeding 100MB) slows down CI/CD pipelines
  4. Bandwidth costs: High-frequency pipelines consume significant bandwidth downloading the same provider versions repeatedly
  5. Rate limiting: External registries may rate-limit requests, causing pipeline failures during high-activity periods

GitLab already provides dependency proxies for container images, PyPI packages, and npm packages, but no equivalent exists for Terraform/OpenTofu providers.

Proposal

Add a Dependency Proxy for Terraform/OpenTofu Providers that:

  1. Proxies and caches provider binaries from upstream registries (registry.terraform.io, registry.opentofu.org)
  2. Implements the Terraform Provider Network Mirror Protocol so it can be used as a provider_installation mirror in Terraform/OpenTofu CLI configuration
  3. Integrates at the group level similar to the existing container Dependency Proxy
  4. Supports authentication using existing GitLab tokens (personal access tokens, CI job tokens, group access tokens)
  5. Provides cache management including storage metrics, cache clearing, and retention policies

Example usage in .terraformrc or CI/CD:

  network_mirror {
    url = "https://gitlab.example.com/api/v4/groups/<group_id>/-/dependency_proxy/terraform/providers/"
  }
}

Existing open-source implementations that could inform the design:

Intended users

  • Priyanka (Platform Engineer) - Managing infrastructure-as-code pipelines and ensuring reliable deployments
  • Sasha (Software Developer) - Using Terraform/OpenTofu in development workflows
  • Sidney (Systems Administrator) - Operating GitLab in air-gapped or network-restricted environments
  • Ingrid (Infrastructure Operator) - Optimizing CI/CD pipeline performance and reducing external dependencies

Feature Usage Metrics

  • Number of groups with Terraform Dependency Proxy enabled
  • Number of provider download requests served from cache vs. upstream
  • Cache hit ratio
  • Total bandwidth saved (cached bytes served)
  • Number of unique providers cached
  • Number of unique users/pipelines using the proxy

Does this feature require an audit event?

Yes, audit events should be considered for:

  • Enabling/disabling the Terraform Dependency Proxy at the group level
  • Cache purge operations
  • Configuration changes to the proxy settings
Edited by 🤖 GitLab Bot 🤖