Reference Terraform modules from a project

What does this MR do and why?

context

When users need to install a terraform module from GitLab's Terraform Registry, they can do so only be providing the namespace path to the module. That means they are limited to a namespace level endpoint to consume the terraform modules.

When GitLab's Terraform Registry receives the module install request, it will search for the requested module in the whole namespace. This forces some limitations on users' experience dealing with the Terraform Registry.

Problem

There is a Terraform limitation concerning namespace: "Module addresses MUST follow the pattern hostname/namespace/name/system".

Due to this limitation namespace has been mapped to the root level sub-group name, so all modules regardless of which project they contributed from, are accessed via the root level namespace, there is no way as a module consumer to know which project this module came from.

This gives us other issues as well in terms of naming collisions for modules that can be contributed from different projects further down the sub-group tree. It also gives us issues in that anybody in the root level tree can add modules to the root level group from any subgroup project. There also seems to be a conceptual mismatch i.e. I add modules from the project level, but I access them via the root level group.

Solution

Terraform provides some alternatives to give us the possibility to deviate from the standard format of the source address hostname/namespace/name/system.

Among the available alternatives, Fetching archives over HTTP is the most suitable for our needs. That means instead of adhering to the hostname/namespace/name/system format to provide the source address, we provide an alternative full HTTP URL. This URL should map to an endpoint in the Terraform Registry which in turn should serve the requested terraform module.

Here's an example of how we can use Fetching archives over HTTP to ask GitLab's Terraform Registry to search for the requested module in the provided project ID:

module "infra_registry_aws" {
  source = "https://gitlab.com/api/v4/projects/1/packages/terraform/modules/infra-registry/aws?archive=tgz"
}

In the provided URL as a source, we provide 4 pieces of information:

• project id => 1
• module name => infra-registry
• module system => aws
• query parameter => archive=tgz. Needed so that Terraform understands that the source URL is the download destination for the module file and it's expected to respond with the module file. Without this query param, Terraform would expect another destination for the file download in the response coming from this URL.

Based on the above, we add in this MR two new project-level endpoints that receive the module download request and respond with the matching module file. One without a module version that would download the latest version; and the other with a specific module version:

• GET /api/v4/projects/<project_id>/packages/terraform/modules/<module_name>/<module_system>
• GET /api/v4/projects/<project_id>/packages/terraform/modules/<module_name>/<module_system>/<module_version>

In this Fetching archives over HTTP source type, Terraform sends the authentication credentials using a .netrc file: https://everything.curl.dev/usingcurl/netrc

Supporting the project-level endpoints to download the module will not replace the current namespace-level implementation. Users would have the liberty to choose how they want to consume their modules.

The project-level source seems verbose compared to the elegant hostname/namespace/name/system format, but this is how we can do it. By deviating from the standard, Terraform wouldn't make any assumptions for us and we need to provide the full URL so it can find the correct module file.

Screenshots or screen recordings

N/A

How to set up and validate locally

1. Make sure you have terraform CLI installed.
2. We need to publish a terraform module to test consuming it. This public project can be used to publish a dummy module.
3. Clone the repo and make sure to modify the required_version to be the same as the installed terraform CLI's version before publishing it.
4. Inside package.sh, replace YOUR_TOKEN, GITLAB_API_V4_URL & PROJECT_ID variable with your local testing values.
5. Navigate to the root of the cloned project and then run the package.sh script file:

./package.sh 

6. The module should be successfully published to your GDK.
7. Create a new directory on your machine, and inside this directory create a new file named main.tf
8. Open the main.tf file and paste the following in it:

# main.tf

module "aws_module" {
  source = "http://gdk.test:3000/api/v4/projects/<project_id>/packages/terraform/modules/infra-registry/aws?archive=tgz"
 
}

9. We need to send our token when sending the above request. To do so, terraform would require the username & token to be stored in a file named .netrc. This file should be stored in a user's home directory ~/.netrc. Open this file and put the following inside it:

machine gdk.test:3000
login <your_username>
password <your_personal_or_deploy_token>

10. In the terminal, run the following command in the root of the new directory:

terraform init

11. The module we published should be downloaded and installed successfully.
12. archive=tgz query param is optional. You can omit it and try referencing the module and things should work the same. However, its presence is recommended to avoid unnecessary redirection.
13. To test installing a specific version, we need to publish the same module but with different versions.
14. As we did in step 4, we can edit package.sh to change the TERRAFORM_MODULE_VERSION constant and then run ./package.sh to publish a new version of the module.
15. Edit main.tf file to include the version in the source URL:

# main.tf

module "aws_module" {
  source = "http://gdk.test:3000/api/v4/projects/<project_id>/packages/terraform/modules/infra-registry/aws/0.0.2?archive=tgz"
 
}

16. Run terraform init & the version 0.0.2 of the module should be installed successfully.

MR acceptance checklist

This checklist encourages us to confirm any changes have been analyzed to reduce risks in quality, performance, reliability, security, and maintainability.

• I have evaluated the MR acceptance checklist for this MR.

Related to #367726 (closed)