Key Improvements for the Terraform module registry
Key Improvements for the Terraform module registry
This page may contain information related to upcoming products, features and functionality. It is important to note that the information presented is for informational purposes only, so please do not rely on the information for purchasing or planning purposes. Just like with all projects, the items mentioned on the page are subject to change or delay, and the development, release, and timing of any products, features, or functionality remain at the sole discretion of GitLab Inc.
Engineering DRI: @mattkasa
Release post candidate
Terraform modules are great to simplify, standardize and speed up infrastructure provisioning inside a company. Until now you could use only file or git based modules with GitLab or an external Terraform registry. By extending the GitLab registry with Terraform modules your modules can be found easily, moreover - as the registry supports semantic versioning - you get simple and secure module updates out of the box for the module's consumers.
Problem to solve
As a DevOps engineer, in order to use versioned modules with Terraform, I'd like to use a GitLab integrated Terraform registry.
Currently, the only convenient way to store private Terraform providers is by using Terraform Cloud without having to write your own service. While this may work for some organizations, I would like to store custom in-house providers on my infrastructure. Since I already use GitLab extensively on my infrastructure, I would like to store the source code for the Terraform provider, the definition of the CI/CD pipeline, and the resulting packing all in the same place.
Intended users
Our own SREs building https://ops.gitlab.net/gitlab-com/gl-infra/terraform-modules
- Sasha (Software Developer)
- Devon (DevOps Engineer)
- Sidney (Systems Administrator)
- Sam (Security Analyst)
Further details
There are many registries out there that we can learn from. Some that are related to infrastructure and configuration:
Proposal
- Under project settings I should be able to tell if my registry should be public or private. For public projects the registry is always public. (This is the same logic what we have with GitLab Pages.)
- We want to support semver versioning in accessing modules.
Iteration 1
- Gitlab supports the Registry protocol required by Terraform https://www.terraform.io/docs/internals/module-registry-protocol.html (gitlab#321102 (closed))
- GitLab supports semantically versioned terraform modules (gitlab#321102 (closed))
-
Terraform can authenticate to Module registry (gitlab#324237 (closed))
- Possible to use https://www.terraform.io/docs/commands/cli-config.html#credentials-1 ?/
-
GitLab List Available Versions for a Specific Module (gitlab#321102 (closed))
- :namespace/:name/:provider/versions
-
Terraform can Download Source Code for a Specific Module Version (gitlab#321102 (closed))
- :namespace/:name/:provider/:system/:version/download
- Add separate documentation page for the Terraform registry (gitlab#327310 (closed))
Iteration 2
- Rollout feature flag gitlab#325064 (closed)
- Remove feature flag gitlab#325065 (closed)
Iteration 3
- Add "Infrastructure Registry" to navigation
- Workhorse? Terraform can discover the registry service https://www.terraform.io/docs/internals/remote-service-discovery.html#discovery-process
How are modules published?
We want to follow Terraform's requirements around modules. This means that a Terraform module can be any git repo with a main.tf
file in its root. Moreover, the GitLab should support the Terraform remote protocols specifications.
A module is published when:
- a
main.tf
file is present in its root - a new semver compliant git tag is added to the repo
In the MVC, we DON'T want to support sub-modules.
Future considerations outside of initial scope
- Terraform sub-module support
- How can we help with module discovery? Once a module is published, it's useful only if people can find it. In the case of Terraform, best practice is to provide an example directory that shows the module in action (and is used for optional tests too). There are tools to generate automatic documentation for a module, do we want to fall back to it if no documentation exists?
- An example repo with tests and following much practice setups: https://gitlab.com/sandlin/aws/terraform_eks/-/tree/master/
Permissions and Security
Documentation
Availability & Testing
What does success look like, and how can we measure that?
What is the type of buyer?
Links / references
Some open source terraform registries:
- Show closed items