Skip to content

Add worker to index terraform module file

Moaz Khalifa requested to merge 438058-index-terraform-module-package into master

What does this MR do and why?

This MR is the second one after Create packages_terraform_module_metadata table... (!141914 - merged) in the effort of implementing Provide module documentation in the Terraform r... (&12408).

At a high level, the implementation of &12408 can be broken down into the following steps:

  1. Creating a new database table to persist the extracted documentation data. (Done in !141914 (merged))
  2. Introduce a background worker that's responsible for triggering a service (or multiple services) to index the terraform module file. Indexing consists of several actions:
    • Extraction of the readme file for the module, its submodules, and its examples.
    • Processing of other files to generate additional documentation data such as inputs, outputs, dependencies & resources.
    • Persistent storage of the readme files and other extracted data in the newly created database table.
  3. Integrate the persisted documentation data into the response sent to the frontend.
  4. Implement the necessary adjustments in the UI to appropriately display the documentation.

This MR is implementing part of step 2: introduce a worker that's responsible for processing the uploaded module archive file, unzip this archive, and extract the files that include the needed metadata from it.

  • Here's a demonstration of the flow:
  %%{init: {"flowchart": {"htmlLabels": false}} }%%
  flowchart TB
    subgraph "In this MR"

    a("Packages::TerraformModule::CreatePackageService") -- "`if **index_terraform_module_archive** feature flag  enabled`" --> b("Packages::TerraformModule::ProcessPackageFileWorker")
    b("Packages::TerraformModule::ProcessPackageFileWorker") --> c("Packages::TerraformModule::ProcessPackageFileService")
    c("Packages::TerraformModule::ProcessPackageFileService") --> d("Packages::TerraformModule::Metadata::ExtractFilesService")
    end
    subgraph "In subsequent MR"
    
    d("Packages::TerraformModule::Metadata::ExtractFilesService") --> e("Packages::TerraformModule::Metadata::ParseFileService")
    d("Packages::TerraformModule::Metadata::ExtractFilesService") --> f("Packages::TerraformModule::Metadata::CreateServie")
    end

For the sake of controlling the size of this MR, Packages::TerraformModule::Metadata::ExtractFilesService is now returning a dummy response instead of the actual correct one. In the subsequent MR, this service should call another 2 services in order to complete the metadata extraction flow, as clarified in the diagram.

Triggering the Packages::TerraformModule::ProcessPackageFileWorker is gated behind the index_terraform_module_archive feature flag. That would allow us to merge this MR safely and then continue working on the subsequent MR that would include the parsing logic. After we merge the complete flow, we can start gradually rollout the feature.

MR acceptance checklist

Please evaluate this MR against the MR acceptance checklist. It helps you analyze changes to reduce risks in quality, performance, reliability, security, and maintainability.

Screenshots or screen recordings

Screenshots are required for UI changes, and strongly recommended for all other merge requests.

Before After

How to set up and validate locally

# stub file upload
def fixture_file_upload(*args, **kwargs)
  Rack::Test::UploadedFile.new(*args, **kwargs)
end

# create a terraform module with a tar archive & zip archive
tar_file = FactoryBot.create(:package_file, :terraform_module)
zip_file = FactoryBot.create(:package_file, :terraform_module, zip: true)

# verify that each archive type can be processed correctly
Packages::TerraformModule::ProcessPackageFileService.new(tar_file).execute
Packages::TerraformModule::ProcessPackageFileService.new(zip_file).execute

# Each service call should print out the extracted files and each file size
#<ServiceResponse:0x000000028505a550
# @http_status=:ok,
# @message=nil,
# @payload={"variables.tf"=>"Size: 254", "README.md"=>"Size: 539", "outputs.tf"=>"Size: 61", "main.tf"=>"Size: 211"},
# @reason=nil,
# @status=:success>

Related to #438058

Edited by Moaz Khalifa

Merge request reports