Add semver to terraform module metadata (!183428) · Merge requests · GitLab.org / GitLab

Context

In Terraform registry: "Latest version for a speci... (#474746 - closed), it was reported that when requesting the latest version of a terraform module, it sometimes doesn't resolve correctly. That's because the sorting is performed in the database on the version string (1.0.3 for example). And unfortunately, the database doesn’t natively understand semantic versions.

To solve this issue, it is necessary to extract the version components to separate columns in the database as explained in this documentation guide.

The problem with terraform modules is that they are saved in the packages_packages table, which hosts all package formats of the package registry. That's why it's not possible to add the new separate SemVer columns to it.

However, there's the packages_terraform_module_metadata database table that we use to persist the README text and other metadata extracted from the uploaded modules. There's a one-to-one association between packages_packages & packages_terraform_module_metadata tables. This would make it join them and sort based on the SemVer columns in the metadata table.

There's an obstacle in the road of the implementation: the metadata table was introduced recently, which means not all terraform modules have a corresponding metadata record. So in order to be able to rely on the metadata table, we have to backfill all terraform modules to make sure they have an existing metadata row with the SemVer new columns populated.

So the implementation plan is:

An MR to add the new SemVer columns to the packages_terraform_module_metadata database table, and start populating them for every new published module. (this MR)
An MR to backfill the SemVer columns for the existing modules that have no corresponding metadata record, or have one, but its SemVer columns are empty.
An MR to replace the current sorting logic with a new SemVer sorting that's available thanks to the SemanticVersionable module.

What does this MR do and why?

Add new SemVer columns to the packages_terraform_module_metadata database table according to this documentation guide.
Populate those columns when publishing any new terraform module in the Terraform Registry.
Add related specs.

References

Please include cross links to any resources that are relevant to this MR. This will give reviewers and future readers helpful context to give an efficient review of the changes introduced.

MR acceptance checklist

Please evaluate this MR against the MR acceptance checklist. It helps you analyze changes to reduce risks in quality, performance, reliability, security, and maintainability.

Screenshots or screen recordings

N/A

How to set up and validate locally

Download this module to test with: infra-registry-aws-0.0.1.tgz
From your terminal, navigate to the directory where the module was downloaded.

Publish the module to your GDK :

 curl --fail-with-body --header "PRIVATE-TOKEN: <PAT>" \
   --upload-file infra-registry-aws-0.0.1.tgz \
   "http://gdk.test:3000/api/v4/projects/<project_id>/packages/terraform/modules/my-module/aws/0.0.1/file"

In rails console, verify that the SemVer columns were populated correctly for the published module:

Packages::TerraformModule::Metadatum.last
=> #<Packages::TerraformModule::Metadatum:0x00000003b31b7c70
  created_at: Tue, 04 Mar 2025 23:43:09.300575000 UTC +00:00,
  updated_at: Tue, 04 Mar 2025 23:43:09.300575000 UTC +00:00,
  package_id: 33,
  project_id: 1,
  fields: ........
  semver_major: 0,
  semver_minor: 0,
  semver_patch: 1,
  semver_prerelease: nil>

You can upload the module with a different version and verify the SemVer columns:

curl --fail-with-body --header "PRIVATE-TOKEN: <PAT>" \
   --upload-file infra-registry-aws-0.0.1.tgz \
   "http://gdk.test:3000/api/v4/projects/<project_id>/packages/terraform/modules/my-module/aws/0.0.9-beta/file"

Related to #474746 (closed)

Edited Mar 05, 2025 by Moaz Khalifa

Add semver to terraform module metadata