Allow skipping Terraform state file encryption based on application setting

What does this MR do and why?

This MR allows skipping encrypting Terraform state files, if the instance setting terraform_state_encryption_enabled is set to false.

Summary of changes:

• Modify uploading logic for Terraform state file to enable skipping encryption depending on the application setting.
• Added new database column is_encrypted to keep track whether the file is encrypted or not.
• Introduced feature flag skip_encrypting_terraform_state_file.

References

Part of Stop encrypting when Terraform state encryption... (#578031).

How to set up and validate locally

• Set up GDK.
• Make sure these two scenarios work as expected.

(1) Current behavior (= Encrypt and decrypt Terraform state file)

Make sure the FF is disabled:

project = Project.find(ID)
Feature.disable(:skip_encrypting_terraform_state_file, project)

Save new Terraform state file using curl:

export PAT=YOUR_PERSONAL_ACCESS_TOKEN

curl -X POST \
  --insecure \
  -H "Authorization: Bearer $PAT" \
  -H "Content-Type: application/json" \
  -d '{"version":5,"terraform_version":"1.0.0","serial":4,"lineage":"test"}' \
  "https://gdk.test:3443/api/v4/projects/40/terraform/state/test-state-1"

Confirm the file is stored with encrypted flag, and Rails is decrypting the file when calling read:

project.terraform_states.last.latest_version.is_encrypted
=> true

> project.terraform_states.last.latest_version.file.read
  Terraform::State Load (0.3ms)  SELECT "terraform_states".* FROM "terraform_states" WHERE "terraform_states"."project_id" = 40 ORDER BY "terraform_states"."id" DESC LIMIT 1 /*application:console,db_config_database:gitlabhq_development,db_config_name:main,console_hostname:tnishida--20230731-6YRFY,console_username:taka,line:(pry):25:in `__pry__'*/
  Terraform::StateVersion Load (0.1ms)  SELECT "terraform_state_versions".* FROM "terraform_state_versions" WHERE "terraform_state_versions"."terraform_state_id" = 27 ORDER BY "terraform_state_versions"."version" DESC LIMIT 1 /*application:console,db_config_database:gitlabhq_development,db_config_name:main,console_hostname:tnishida--20230731-6YRFY,console_username:taka,line:(pry):25:in `__pry__'*/
  Decrypt File (5.2ms) Decrypted file # <====== WE CAN CONFIRM IT'S DECRYPTING THE FILE =========
=> "{\"version\":5,\"terraform_version\":\"1.0.0\",\"serial\":4,\"lineage\":\"test\"}"

project.terraform_states.last.latest_version.file.path
=> "/Users/taka/gitlab-development-kit/gitlab/shared/terraform_state/d5/9e/d59eced1ded07f84c145592f65bdf854358e009c5cd705f5215bf18697fed103/2bac2613f4f9d9e1852d5fbf52246940/4.tfstate"

Confirm that the stored file is encrypted:

$ cat /Users/taka/gitlab-development-kit/gitlab/shared/terraform_state/d5/9e/d59eced1ded07f84c145592f65bdf854358e009c5cd705f5215bf18697fed103/2bac2613f4f9d9e1852d5fbf52246940/4.tfstate
7�-�pD�hºY*�!�#yJh�9G�����r���]S�:����2YI�ba*��)���~�?��N40]`��^�1� �à�E�(�%

(2) New behavior: Enable skipping encryption for Terraform state file

Make sure the FF is enabled AND the application setting terraform_state_encryption_enabled is set to false:

project = Project.find(ID)
Feature.enable(:skip_encrypting_terraform_state_file, project)
ApplicationSetting.current.update!(terraform_state_encryption_enabled: false)

Save new Terraform state file:

export PAT=YOUR_PERSONAL_ACCESS_TOKEN

curl -X POST \
  --insecure \
  -H "Authorization: Bearer $PAT" \
  -H "Content-Type: application/json" \
  -d '{"version":5,"terraform_version":"1.0.0","serial":4,"lineage":"test"}' \
  "https://gdk.test:3443/api/v4/projects/40/terraform/state/test-state-1"

Confirm the file is stored with encrypted flag = false, and there are no decryption logs:

project.terraform_states.last.latest_version.is_encrypted?
=> false

# There are no decryption log:
project.terraform_states.last.latest_version.file.read
  Terraform::State Load (1.4ms)  SELECT "terraform_states".* FROM "terraform_states" WHERE "terraform_states"."project_id" = 40 ORDER BY "terraform_states"."id" DESC LIMIT 1 /*application:console,db_config_database:gitlabhq_development,db_config_name:main,console_hostname:tnishida--20230731-6YRFY,console_username:taka,line:(pry):22:in `__pry__'*/
  Terraform::StateVersion Load (0.5ms)  SELECT "terraform_state_versions".* FROM "terraform_state_versions" WHERE "terraform_state_versions"."terraform_state_id" = 27 ORDER BY "terraform_state_versions"."version" DESC LIMIT 1 /*application:console,db_config_database:gitlabhq_development,db_config_name:main,console_hostname:tnishida--20230731-6YRFY,console_username:taka,line:(pry):22:in `__pry__'*/
=> "{\"version\":5,\"terraform_version\":\"1.0.0\",\"serial\":3,\"lineage\":\"test\"}"

project.terraform_states.last.latest_version.file.path
=> "/Users/taka/gitlab-development-kit/gitlab/shared/terraform_state/d5/9e/d59eced1ded07f84c145592f65bdf854358e009c5cd705f5215bf18697fed103/2bac2613f4f9d9e1852d5fbf52246940/3.tfstate"

Confirm the file is actually NOT encrypted:

$ cat /Users/taka/gitlab-development-kit/gitlab/shared/terraform_state/d5/9e/d59eced1ded07f84c145592f65bdf854358e009c5cd705f5215bf18697fed103/2bac2613f4f9d9e1852d5fbf52246940/3.tfstate
{"version":5,"terraform_version":"1.0.0","serial":3,"lineage":"test"}%

MR acceptance checklist

Evaluate this MR against the MR acceptance checklist. It helps you analyze changes to reduce risks in quality, performance, reliability, security, and maintainability.