Add support for replicating terraform states
What does this MR do?
This merge request adds geo replication to Terraform::State
, feature flagged off by default.
Screenshots
For Database Review
terraform_states
as of 2020-07-10 15:00Z has 378 rows.
Migration output:
$ bin/rails db:migrate RAILS_ENV=test
== 20200710152642 AddVerificationStateToTerraformStates: migrating ============
-- change_table(:terraform_states)
-> 0.0049s
== 20200710152642 AddVerificationStateToTerraformStates: migrated (0.0050s) ===
== 20200710153009 AddVerificationFailureLimitAndIndexToTerraformStates: migrating
-- transaction_open?()
-> 0.0000s
-- index_exists?(:terraform_states, :verification_failure, {:where=>"(verification_failure IS NOT NULL)", :name=>"terraform_states_verification_failure_partial", :algorithm=>:concurrently})
-> 0.0015s
-- add_index(:terraform_states, :verification_failure, {:where=>"(verification_failure IS NOT NULL)", :name=>"terraform_states_verification_failure_partial", :algorithm=>:concurrently})
-> 0.0021s
-- transaction_open?()
-> 0.0000s
-- index_exists?(:terraform_states, :verification_checksum, {:where=>"(verification_checksum IS NOT NULL)", :name=>"terraform_states_verification_checksum_partial", :algorithm=>:concurrently})
-> 0.0016s
-- add_index(:terraform_states, :verification_checksum, {:where=>"(verification_checksum IS NOT NULL)", :name=>"terraform_states_verification_checksum_partial", :algorithm=>:concurrently})
-> 0.0020s
-- transaction_open?()
-> 0.0000s
-- execute("ALTER TABLE terraform_states\nADD CONSTRAINT check_21a47163ea\nCHECK ( char_length(verification_failure) <= 255 )\nNOT VALID;\n")
-> 0.0006s
-- execute("ALTER TABLE terraform_states VALIDATE CONSTRAINT check_21a47163ea;")
-> 0.0007s
== 20200710153009 AddVerificationFailureLimitAndIndexToTerraformStates: migrated (0.0136s)
Rails Rollback:
== 20200710153009 AddVerificationFailureLimitAndIndexToTerraformStates: reverting
-- transaction_open?()
-> 0.0000s
-- index_exists?(:terraform_states, :verification_failure, {:algorithm=>:concurrently})
-> 0.0074s
-- remove_index(:terraform_states, {:algorithm=>:concurrently, :column=>:verification_failure})
-> 0.0144s
-- transaction_open?()
-> 0.0000s
-- index_exists?(:terraform_states, :verification_checksum, {:algorithm=>:concurrently})
-> 0.0016s
-- remove_index(:terraform_states, {:algorithm=>:concurrently, :column=>:verification_checksum})
-> 0.0038s
-- execute("ALTER TABLE terraform_states\nDROP CONSTRAINT IF EXISTS check_21a47163ea\n")
-> 0.0011s
== 20200710153009 AddVerificationFailureLimitAndIndexToTerraformStates: reverted (0.0355s)
== 20200710152642 AddVerificationStateToTerraformStates: reverting ============
-- remove_column(:terraform_states, :verification_failure, :text, {})
-> 0.0026s
-- remove_column(:terraform_states, :verification_checksum, :binary, {:using=>"verification_checksum::bytea"})
-> 0.0004s
-- remove_column(:terraform_states, :verification_retry_count, :integer, {:limit=>2})
-> 0.0002s
-- remove_column(:terraform_states, :verified_at, :datetime_with_timezone, {})
-> 0.0002s
-- remove_column(:terraform_states, :verification_retry_at, :datetime_with_timezone, {})
-> 0.0016s
== 20200710152642 AddVerificationStateToTerraformStates: reverted (0.0079s) ===
Geo Migration:
== 20200707210300 CreateTerraformStateRegistry: migrating =====================
-- table_exists?(:terraform_state_registry)
-> 0.0005s
-- create_table(:terraform_state_registry, {:id=>:bigserial, :force=>:cascade})
-> 0.0126s
-- transaction_open?()
-> 0.0000s
-- execute("ALTER TABLE terraform_state_registry\nADD CONSTRAINT check_70a3f43f16\nCHECK ( char_length(last_sync_failure) <= 255 )\nNOT VALID;\n")
-> 0.0006s
-- execute("ALTER TABLE terraform_state_registry VALIDATE CONSTRAINT check_70a3f43f16;")
-> 0.0008s
== 20200707210300 CreateTerraformStateRegistry: migrated (0.0216s) ============
Geo Rollback:
== 20200707210300 CreateTerraformStateRegistry: reverting =====================
-- drop_table(:terraform_state_registry)
-> 0.0220s
== 20200707210300 CreateTerraformStateRegistry: reverted (0.0221s) ============
Does this MR meet the acceptance criteria?
Conformity
-
Changelog entry -
Documentation (if required) -
Code review guidelines -
Merge request performance guidelines -
Style guides -
Database guides -
Separation of EE specific content
Availability and Testing
-
Review and add/update tests for this feature/bug. Consider all test levels. See the Test Planning Process. -
Tested in all supported browsers -
Informed Infrastructure department of a default or new setting change, if applicable per definition of done
Manual Local Testing steps (For posterity):
- Start primary and secondary gdk geos
- Enable the feature flag
bundle exec rails runner 'Feature.enable(:geo_terraform_state_replication)'
- Generate an api key for a user
- Create a project to use for terraform that the api user has maintainer access to
- Create a terraform file
main.tf
terraform { backend "http" { } }
- Run:
$ export USERNAME={gdk user} $ export TOKEN={gdk token} $ export NAMESPACE={namespace%2Aproject, ex: flightjs%2Fflight} $ terraform init \ -backend-config="address=https://gdk.test:3443/api/v4/projects/$NAMESPACE/terraform/state/test" \ -backend-config="lock_address=https://gdk.test:3443/api/v4/projects/$NAMESPACE/terraform/state/test/lock" \ -backend-config="unlock_address=https://gdk.test:3443/api/v4/projects/$NAMESPACE/terraform/state/test/lock" \ -backend-config="username=$USERNAME" \ -backend-config="password=$TOKEN" \ -backend-config="lock_method=POST" \ -backend-config="unlock_method=DELETE" \ -backend-config="retry_wait_min=5"
- Validate that the state shows up at
gdk/gitlab/shared/terraform_state/
(it'll be encrypted, but the file should be there) - Validate that the state shows up in the same place on your geo secondary
Relates to #220953 (closed)
Edited by Alex Ives