Add support for replicating terraform states

What does this MR do?

This merge request adds geo replication to Terraform::State, feature flagged off by default.

Screenshots

For Database Review

terraform_states as of 2020-07-10 15:00Z has 378 rows.

Migration output:

$ bin/rails db:migrate RAILS_ENV=test

== 20200710152642 AddVerificationStateToTerraformStates: migrating ============
-- change_table(:terraform_states)
   -> 0.0049s
== 20200710152642 AddVerificationStateToTerraformStates: migrated (0.0050s) ===

== 20200710153009 AddVerificationFailureLimitAndIndexToTerraformStates: migrating 
-- transaction_open?()
   -> 0.0000s
-- index_exists?(:terraform_states, :verification_failure, {:where=>"(verification_failure IS NOT NULL)", :name=>"terraform_states_verification_failure_partial", :algorithm=>:concurrently})
   -> 0.0015s
-- add_index(:terraform_states, :verification_failure, {:where=>"(verification_failure IS NOT NULL)", :name=>"terraform_states_verification_failure_partial", :algorithm=>:concurrently})
   -> 0.0021s
-- transaction_open?()
   -> 0.0000s
-- index_exists?(:terraform_states, :verification_checksum, {:where=>"(verification_checksum IS NOT NULL)", :name=>"terraform_states_verification_checksum_partial", :algorithm=>:concurrently})
   -> 0.0016s
-- add_index(:terraform_states, :verification_checksum, {:where=>"(verification_checksum IS NOT NULL)", :name=>"terraform_states_verification_checksum_partial", :algorithm=>:concurrently})
   -> 0.0020s
-- transaction_open?()
   -> 0.0000s
-- execute("ALTER TABLE terraform_states\nADD CONSTRAINT check_21a47163ea\nCHECK ( char_length(verification_failure) <= 255 )\nNOT VALID;\n")
   -> 0.0006s
-- execute("ALTER TABLE terraform_states VALIDATE CONSTRAINT check_21a47163ea;")
   -> 0.0007s
== 20200710153009 AddVerificationFailureLimitAndIndexToTerraformStates: migrated (0.0136s) 

Rails Rollback:

== 20200710153009 AddVerificationFailureLimitAndIndexToTerraformStates: reverting 
-- transaction_open?()
   -> 0.0000s
-- index_exists?(:terraform_states, :verification_failure, {:algorithm=>:concurrently})
   -> 0.0074s
-- remove_index(:terraform_states, {:algorithm=>:concurrently, :column=>:verification_failure})
   -> 0.0144s
-- transaction_open?()
   -> 0.0000s
-- index_exists?(:terraform_states, :verification_checksum, {:algorithm=>:concurrently})
   -> 0.0016s
-- remove_index(:terraform_states, {:algorithm=>:concurrently, :column=>:verification_checksum})
   -> 0.0038s
-- execute("ALTER TABLE terraform_states\nDROP CONSTRAINT IF EXISTS check_21a47163ea\n")
   -> 0.0011s
== 20200710153009 AddVerificationFailureLimitAndIndexToTerraformStates: reverted (0.0355s) 

== 20200710152642 AddVerificationStateToTerraformStates: reverting ============
-- remove_column(:terraform_states, :verification_failure, :text, {})
   -> 0.0026s
-- remove_column(:terraform_states, :verification_checksum, :binary, {:using=>"verification_checksum::bytea"})
   -> 0.0004s
-- remove_column(:terraform_states, :verification_retry_count, :integer, {:limit=>2})
   -> 0.0002s
-- remove_column(:terraform_states, :verified_at, :datetime_with_timezone, {})
   -> 0.0002s
-- remove_column(:terraform_states, :verification_retry_at, :datetime_with_timezone, {})
   -> 0.0016s
== 20200710152642 AddVerificationStateToTerraformStates: reverted (0.0079s) ===

Geo Migration:

== 20200707210300 CreateTerraformStateRegistry: migrating =====================
-- table_exists?(:terraform_state_registry)
   -> 0.0005s
-- create_table(:terraform_state_registry, {:id=>:bigserial, :force=>:cascade})
   -> 0.0126s
-- transaction_open?()
   -> 0.0000s
-- execute("ALTER TABLE terraform_state_registry\nADD CONSTRAINT check_70a3f43f16\nCHECK ( char_length(last_sync_failure) <= 255 )\nNOT VALID;\n")
   -> 0.0006s
-- execute("ALTER TABLE terraform_state_registry VALIDATE CONSTRAINT check_70a3f43f16;")
   -> 0.0008s
== 20200707210300 CreateTerraformStateRegistry: migrated (0.0216s) ============

Geo Rollback:

== 20200707210300 CreateTerraformStateRegistry: reverting =====================
-- drop_table(:terraform_state_registry)
   -> 0.0220s
== 20200707210300 CreateTerraformStateRegistry: reverted (0.0221s) ============

Does this MR meet the acceptance criteria?

Conformity

• Changelog entry
• Documentation (if required)
• Code review guidelines
• Merge request performance guidelines
• Style guides
• Database guides
• Separation of EE specific content

Availability and Testing

• Review and add/update tests for this feature/bug. Consider all test levels. See the Test Planning Process.
• Tested in all supported browsers
• Informed Infrastructure department of a default or new setting change, if applicable per definition of done

Manual Local Testing steps (For posterity):

1. Start primary and secondary gdk geos
2. Enable the feature flag bundle exec rails runner 'Feature.enable(:geo_terraform_state_replication)'
3. Generate an api key for a user
4. Create a project to use for terraform that the api user has maintainer access to
5. Create a terraform file main.tf

   terraform {
     backend "http" {
     }
   }

6. Run:

   $ export USERNAME={gdk user}
   $ export TOKEN={gdk token}
   $ export NAMESPACE={namespace%2Aproject, ex: flightjs%2Fflight}
   $ terraform init \
     -backend-config="address=https://gdk.test:3443/api/v4/projects/$NAMESPACE/terraform/state/test" \
     -backend-config="lock_address=https://gdk.test:3443/api/v4/projects/$NAMESPACE/terraform/state/test/lock" \
     -backend-config="unlock_address=https://gdk.test:3443/api/v4/projects/$NAMESPACE/terraform/state/test/lock" \
     -backend-config="username=$USERNAME" \
     -backend-config="password=$TOKEN" \
     -backend-config="lock_method=POST" \
     -backend-config="unlock_method=DELETE" \
     -backend-config="retry_wait_min=5"

7. Validate that the state shows up at gdk/gitlab/shared/terraform_state/ (it'll be encrypted, but the file should be there)
8. Validate that the state shows up in the same place on your geo secondary

Relates to #220953 (closed)