500 Error on Terraform State push/pull

Summary

500 Error when trying to push terraform state to gitlab's terraform state backend. When trying manually from my own machine I get error 500: Checking logs I see:

ActiveRecord::RecordNotUnique (PG::UniqueViolation: ERROR:  duplicate key value violates unique constraint "index_terraform_state_versions_on_state_id_and_version"
DETAIL:  Key (terraform_state_id, version)=(1, 0) already exists.

Steps to reproduce

  1. Run a terraform project with the backend set to a gitlab repository; delete a statefile
  2. Create a new terraform project with a gitlab backend, do not delete this statefile
  3. try downloading the statefile as json (I'm not sure this is relevant, but that json file did return 500)
  4. ???
  5. At some point it stops working.

NOTE: this is a new project, not subject to any legacy or upgrades.

What is the current bug behavior?

500 on Sync to terraform backend

2021/11/02 19:01:59 [DEBUG] POST https://gitlab.sharkmob.cloud/api/v4/projects/18/terraform/state/dev-init/lock
2021/11/02 19:01:59 [DEBUG] GET https://gitlab.sharkmob.cloud/api/v4/projects/18/terraform/state/dev-init
2021/11/02 19:02:02 [DEBUG] GET https://gitlab.sharkmob.cloud/api/v4/projects/18/terraform/state/dev-init
2021/11/02 19:02:02 [DEBUG] POST https://gitlab.sharkmob.cloud/api/v4/projects/18/terraform/state/dev-init?ID=45dc6a52-3c88-b390-52de-e5dd32f423d8
2021/11/02 19:02:03 [DEBUG] POST https://gitlab.sharkmob.cloud/api/v4/projects/18/terraform/state/dev-init?ID=45dc6a52-3c88-b390-52de-e5dd32f423d8 (status: 500): retrying in 5s (2 left)
2021/11/02 19:02:08 [DEBUG] POST https://gitlab.sharkmob.cloud/api/v4/projects/18/terraform/state/dev-init?ID=45dc6a52-3c88-b390-52de-e5dd32f423d8 (status: 500): retrying in 10s (1 left)
╷
│ Warning: Empty or non-existent state
│ 
│ There are currently no resources tracked in the state, so there is nothing to refresh.
╵
╷
│ Error: Failed to write state: Failed to upload state: POST https://gitlab.sharkmob.cloud/api/v4/projects/18/terraform/state/dev-init?ID=45dc6a52-3c88-b390-52de-e5dd32f423d8 giving up after 3 attempts
│ 
│ 
╵
2021/11/02 19:02:18 [DEBUG] DELETE https://gitlab.sharkmob.cloud/api/v4/projects/18/terraform/state/dev-init/lock

What is the expected correct behavior?

backend should return 200 and store my terraform state.

Relevant logs and/or screenshots

Completed 200 OK in 441ms (Views: 0.2ms | ActiveRecord: 39.4ms | Elasticsearch: 0.0ms | Allocations: 99928)
ActiveRecord::RecordNotUnique (PG::UniqueViolation: ERROR:  duplicate key value violates unique constraint "index_terraform_state_versions_on_state_id_and_version"
DETAIL:  Key (terraform_state_id, version)=(1, 0) already exists.
):
  /srv/gitlab/vendor/bundle/ruby/2.7.0/gems/activerecord-6.1.4.1/lib/active_record/connection_adapters/postgresql_adapter.rb:672:in `exec_params'
  /srv/gitlab/vendor/bundle/ruby/2.7.0/gems/activerecord-6.1.4.1/lib/active_record/connection_adapters/postgresql_adapter.rb:672:in `block (2 levels) in exec_no_cache'
  /srv/gitlab/vendor/bundle/ruby/2.7.0/gems/activesupport-6.1.4.1/lib/active_support/dependencies/interlock.rb:48:in `block in permit_concurrent_loads'
  /srv/gitlab/vendor/bundle/ruby/2.7.0/gems/activesupport-6.1.4.1/lib/active_support/concurrency/share_lock.rb:187:in `yield_shares'
  /srv/gitlab/vendor/bundle/ruby/2.7.0/gems/activesupport-6.1.4.1/lib/active_support/dependencies/interlock.rb:47:in `permit_concurrent_loads'
  /srv/gitlab/vendor/bundle/ruby/2.7.0/gems/activerecord-6.1.4.1/lib/active_record/connection_adapters/postgresql_adapter.rb:671:in `block in exec_no_cache'
  /srv/gitlab/vendor/bundle/ruby/2.7.0/gems/activerecord-6.1.4.1/lib/active_record/connection_adapters/abstract_adapter.rb:696:in `block (2 levels) in log'
  /srv/gitlab/vendor/bundle/ruby/2.7.0/gems/activesupport-6.1.4.1/lib/active_support/concurrency/load_interlock_aware_monitor.rb:26:in `block (2 levels) in synchronize'
  /srv/gitlab/vendor/bundle/ruby/2.7.0/gems/activesupport-6.1.4.1/lib/active_support/concurrency/load_interlock_aware_monitor.rb:25:in `handle_interrupt'
  /srv/gitlab/vendor/bundle/ruby/2.7.0/gems/activesupport-6.1.4.1/lib/active_support/concurrency/load_interlock_aware_monitor.rb:25:in `block in synchronize'
  /srv/gitlab/vendor/bundle/ruby/2.7.0/gems/activesupport-6.1.4.1/lib/active_support/concurrency/load_interlock_aware_monitor.rb:21:in `handle_interrupt'
  /srv/gitlab/vendor/bundle/ruby/2.7.0/gems/activesupport-6.1.4.1/lib/active_support/concurrency/load_interlock_aware_monitor.rb:21:in `synchronize'
  /srv/gitlab/vendor/bundle/ruby/2.7.0/gems/activerecord-6.1.4.1/lib/active_record/connection_adapters/abstract_adapter.rb:695:in `block in log'
  /srv/gitlab/vendor/bundle/ruby/2.7.0/gems/activesupport-6.1.4.1/lib/active_support/notifications/instrumenter.rb:24:in `instrument'
  /srv/gitlab/vendor/bundle/ruby/2.7.0/gems/activerecord-6.1.4.1/lib/active_record/connection_adapters/abstract_adapter.rb:687:in `log'
  /srv/gitlab/vendor/bundle/ruby/2.7.0/gems/activerecord-6.1.4.1/lib/active_record/connection_adapters/postgresql_adapter.rb:670:in `exec_no_cache'
  /srv/gitlab/vendor/bundle/ruby/2.7.0/gems/activerecord-6.1.4.1/lib/active_record/connection_adapters/postgresql_adapter.rb:649:in `execute_and_clear'
  /srv/gitlab/vendor/bundle/ruby/2.7.0/gems/marginalia-1.10.0/lib/marginalia.rb:95:in `execute_and_clear_with_marginalia'
  /srv/gitlab/vendor/bundle/ruby/2.7.0/gems/activerecord-6.1.4.1/lib/active_record/connection_adapters/postgresql/database_statements.rb:53:in `exec_query'
  /srv/gitlab/vendor/bundle/ruby/2.7.0/gems/activerecord-6.1.4.1/lib/active_record/connection_adapters/abstract/database_statements.rb:136:in `exec_insert'
  /srv/gitlab/vendor/bundle/ruby/2.7.0/gems/activerecord-6.1.4.1/lib/active_record/connection_adapters/postgresql/database_statements.rb:91:in `exec_insert'
  /srv/gitlab/vendor/bundle/ruby/2.7.0/gems/activerecord-6.1.4.1/lib/active_record/connection_adapters/abstract/database_statements.rb:171:in `insert'
  /srv/gitlab/vendor/bundle/ruby/2.7.0/gems/activerecord-6.1.4.1/lib/active_record/connection_adapters/abstract/query_cache.rb:22:in `insert'
  /srv/gitlab/lib/gitlab/database/load_balancing/connection_proxy.rb:119:in `block in write_using_load_balancer'
  /srv/gitlab/lib/gitlab/database/load_balancing/load_balancer.rb:107:in `block in read_write'
  /srv/gitlab/lib/gitlab/database/load_balancing/load_balancer.rb:167:in `retry_with_backoff'
  /srv/gitlab/lib/gitlab/database/load_balancing/load_balancer.rb:105:in `read_write'
  /srv/gitlab/lib/gitlab/database/load_balancing/connection_proxy.rb:118:in `write_using_load_balancer'
  /srv/gitlab/lib/gitlab/database/load_balancing/connection_proxy.rb:60:in `block (2 levels) in <class:ConnectionProxy>'
  /srv/gitlab/vendor/bundle/ruby/2.7.0/gems/activerecord-6.1.4.1/lib/active_record/persistence.rb:375:in `_insert_record'
  /srv/gitlab/vendor/bundle/ruby/2.7.0/gems/activerecord-6.1.4.1/lib/active_record/persistence.rb:929:in `_create_record'
  /srv/gitlab/vendor/bundle/ruby/2.7.0/gems/activerecord-6.1.4.1/lib/active_record/counter_cache.rb:166:in `_create_record'
  /srv/gitlab/vendor/bundle/ruby/2.7.0/gems/activerecord-6.1.4.1/lib/active_record/locking/optimistic.rb:79:in `_create_record'
  /srv/gitlab/vendor/bundle/ruby/2.7.0/gems/activerecord-6.1.4.1/lib/active_record/attribute_methods/dirty.rb:201:in `_create_record'
  /srv/gitlab/vendor/bundle/ruby/2.7.0/gems/activerecord-6.1.4.1/lib/active_record/callbacks.rb:461:in `block in _create_record'
  /srv/gitlab/vendor/bundle/ruby/2.7.0/gems/activesupport-6.1.4.1/lib/active_support/callbacks.rb:98:in `run_callbacks'
  /srv/gitlab/vendor/bundle/ruby/2.7.0/gems/activesupport-6.1.4.1/lib/active_support/callbacks.rb:824:in `_run_create_callbacks'
  /srv/gitlab/vendor/bundle/ruby/2.7.0/gems/activerecord-6.1.4.1/lib/active_record/callbacks.rb:461:in `_create_record'
  /srv/gitlab/vendor/bundle/ruby/2.7.0/gems/activerecord-6.1.4.1/lib/active_record/timestamp.rb:108:in `_create_record'
  /srv/gitlab/vendor/bundle/ruby/2.7.0/gems/activerecord-6.1.4.1/lib/active_record/persistence.rb:900:in `create_or_update'
  /srv/gitlab/vendor/bundle/ruby/2.7.0/gems/activerecord-6.1.4.1/lib/active_record/callbacks.rb:457:in `block in create_or_update'
  /srv/gitlab/vendor/bundle/ruby/2.7.0/gems/activesupport-6.1.4.1/lib/active_support/callbacks.rb:106:in `run_callbacks'
  /srv/gitlab/vendor/bundle/ruby/2.7.0/gems/activesupport-6.1.4.1/lib/active_support/callbacks.rb:824:in `_run_save_callbacks'
  /srv/gitlab/vendor/bundle/ruby/2.7.0/gems/activerecord-6.1.4.1/lib/active_record/callbacks.rb:457:in `create_or_update'
  /srv/gitlab/vendor/bundle/ruby/2.7.0/gems/activerecord-6.1.4.1/lib/active_record/timestamp.rb:126:in `create_or_update'
  /srv/gitlab/vendor/bundle/ruby/2.7.0/gems/activerecord-6.1.4.1/lib/active_record/persistence.rb:507:in `save!'
  /srv/gitlab/vendor/bundle/ruby/2.7.0/gems/activerecord-6.1.4.1/lib/active_record/validations.rb:53:in `save!'
  /srv/gitlab/vendor/bundle/ruby/2.7.0/gems/activerecord-6.1.4.1/lib/active_record/transactions.rb:302:in `block in save!'
  /srv/gitlab/vendor/bundle/ruby/2.7.0/gems/activerecord-6.1.4.1/lib/active_record/transactions.rb:354:in `block in with_transaction_returning_status'
  /srv/gitlab/vendor/bundle/ruby/2.7.0/gems/activerecord-6.1.4.1/lib/active_record/connection_adapters/abstract/database_statements.rb:318:in `transaction'
  /srv/gitlab/lib/gitlab/database/load_balancing/connection_proxy.rb:119:in `block in write_using_load_balancer'
  /srv/gitlab/lib/gitlab/database/load_balancing/load_balancer.rb:107:in `block in read_write'
  /srv/gitlab/lib/gitlab/database/load_balancing/load_balancer.rb:167:in `retry_with_backoff'
  /srv/gitlab/lib/gitlab/database/load_balancing/load_balancer.rb:105:in `read_write'
  /srv/gitlab/lib/gitlab/database/load_balancing/connection_proxy.rb:118:in `write_using_load_balancer'
  /srv/gitlab/lib/gitlab/database/load_balancing/connection_proxy.rb:70:in `transaction'
  /srv/gitlab/vendor/bundle/ruby/2.7.0/gems/activerecord-6.1.4.1/lib/active_record/transactions.rb:350:in `with_transaction_returning_status'
  /srv/gitlab/vendor/bundle/ruby/2.7.0/gems/activerecord-6.1.4.1/lib/active_record/transactions.rb:302:in `save!'
  /srv/gitlab/vendor/bundle/ruby/2.7.0/gems/activerecord-6.1.4.1/lib/active_record/suppressor.rb:48:in `save!'
  /srv/gitlab/vendor/bundle/ruby/2.7.0/gems/state_machines-activerecord-0.8.0/lib/state_machines/integrations/active_record.rb:515:in `block in save!'
  /srv/gitlab/vendor/bundle/ruby/2.7.0/gems/state_machines-activerecord-0.8.0/lib/state_machines/integrations/active_record.rb:530:in `block in around_save'
  /srv/gitlab/vendor/bundle/ruby/2.7.0/gems/state_machines-0.5.0/lib/state_machines/transition_collection.rb:150:in `block in run_actions'
  /srv/gitlab/vendor/bundle/ruby/2.7.0/gems/state_machines-0.5.0/lib/state_machines/transition_collection.rb:170:in `catch_exceptions'
  /srv/gitlab/vendor/bundle/ruby/2.7.0/gems/state_machines-0.5.0/lib/state_machines/transition_collection.rb:148:in `run_actions'
  /srv/gitlab/vendor/bundle/ruby/2.7.0/gems/state_machines-0.5.0/lib/state_machines/transition_collection.rb:133:in `run_callbacks'
  /srv/gitlab/vendor/bundle/ruby/2.7.0/gems/state_machines-0.5.0/lib/state_machines/transition_collection.rb:212:in `run_callbacks'
  /srv/gitlab/vendor/bundle/ruby/2.7.0/gems/state_machines-0.5.0/lib/state_machines/transition_collection.rb:63:in `block (2 levels) in perform'
  /srv/gitlab/vendor/bundle/ruby/2.7.0/gems/state_machines-0.5.0/lib/state_machines/transition_collection.rb:63:in `catch'
  /srv/gitlab/vendor/bundle/ruby/2.7.0/gems/state_machines-0.5.0/lib/state_machines/transition_collection.rb:63:in `block in perform'
  /srv/gitlab/vendor/bundle/ruby/2.7.0/gems/state_machines-0.5.0/lib/state_machines/transition_collection.rb:186:in `within_transaction'
  /srv/gitlab/vendor/bundle/ruby/2.7.0/gems/state_machines-0.5.0/lib/state_machines/transition_collection.rb:62:in `perform'
  /srv/gitlab/vendor/bundle/ruby/2.7.0/gems/state_machines-activerecord-0.8.0/lib/state_machines/integrations/active_record.rb:530:in `around_save'
  /srv/gitlab/vendor/bundle/ruby/2.7.0/gems/state_machines-activerecord-0.8.0/lib/state_machines/integrations/active_record.rb:515:in `save!'
  /srv/gitlab/app/models/terraform/state.rb:90:in `create_new_version!'
  /srv/gitlab/app/models/terraform/state.rb:49:in `update_file!'
  /srv/gitlab/lib/api/terraform/state.rb:58:in `block (4 levels) in <class:State>'
  /srv/gitlab/app/services/terraform/remote_state_handler.rb:20:in `block in handle_with_lock'
  /srv/gitlab/app/services/terraform/remote_state_handler.rb:61:in `block (2 levels) in retrieve_with_lock'
  /srv/gitlab/vendor/bundle/ruby/2.7.0/gems/activerecord-6.1.4.1/lib/active_record/locking/pessimistic.rb:88:in `block in with_lock'
  /srv/gitlab/vendor/bundle/ruby/2.7.0/gems/activerecord-6.1.4.1/lib/active_record/connection_adapters/abstract/database_statements.rb:320:in `block in transaction'
  /srv/gitlab/vendor/bundle/ruby/2.7.0/gems/activerecord-6.1.4.1/lib/active_record/connection_adapters/abstract/transaction.rb:319:in `block in within_new_transaction'
  /srv/gitlab/vendor/bundle/ruby/2.7.0/gems/activesupport-6.1.4.1/lib/active_support/concurrency/load_interlock_aware_monitor.rb:26:in `block (2 levels) in synchronize'
  /srv/gitlab/vendor/bundle/ruby/2.7.0/gems/activesupport-6.1.4.1/lib/active_support/concurrency/load_interlock_aware_monitor.rb:25:in `handle_interrupt'
  /srv/gitlab/vendor/bundle/ruby/2.7.0/gems/activesupport-6.1.4.1/lib/active_support/concurrency/load_interlock_aware_monitor.rb:25:in `block in synchronize'
  /srv/gitlab/vendor/bundle/ruby/2.7.0/gems/activesupport-6.1.4.1/lib/active_support/concurrency/load_interlock_aware_monitor.rb:21:in `handle_interrupt'
  /srv/gitlab/vendor/bundle/ruby/2.7.0/gems/activesupport-6.1.4.1/lib/active_support/concurrency/load_interlock_aware_monitor.rb:21:in `synchronize'
  /srv/gitlab/vendor/bundle/ruby/2.7.0/gems/activerecord-6.1.4.1/lib/active_record/connection_adapters/abstract/transaction.rb:317:in `within_new_transaction'
  /srv/gitlab/vendor/bundle/ruby/2.7.0/gems/activerecord-6.1.4.1/lib/active_record/connection_adapters/abstract/database_statements.rb:320:in `transaction'
  /srv/gitlab/lib/gitlab/database/load_balancing/connection_proxy.rb:119:in `block in write_using_load_balancer'
  /srv/gitlab/lib/gitlab/database/load_balancing/load_balancer.rb:107:in `block in read_write'
  /srv/gitlab/lib/gitlab/database/load_balancing/load_balancer.rb:167:in `retry_with_backoff'
  /srv/gitlab/lib/gitlab/database/load_balancing/load_balancer.rb:105:in `read_write'
  /srv/gitlab/lib/gitlab/database/load_balancing/connection_proxy.rb:118:in `write_using_load_balancer'
  /srv/gitlab/lib/gitlab/database/load_balancing/connection_proxy.rb:70:in `transaction'
  /srv/gitlab/vendor/bundle/ruby/2.7.0/gems/activerecord-6.1.4.1/lib/active_record/transactions.rb:209:in `transaction'
  /srv/gitlab/lib/gitlab/database.rb:271:in `block in transaction'
  /srv/gitlab/vendor/bundle/ruby/2.7.0/gems/activesupport-6.1.4.1/lib/active_support/notifications.rb:203:in `block in instrument'
  /srv/gitlab/vendor/bundle/ruby/2.7.0/gems/activesupport-6.1.4.1/lib/active_support/notifications/instrumenter.rb:24:in `instrument'
  /srv/gitlab/vendor/bundle/ruby/2.7.0/gems/activesupport-6.1.4.1/lib/active_support/notifications.rb:203:in `instrument'
  /srv/gitlab/lib/gitlab/database.rb:270:in `transaction'
  /srv/gitlab/vendor/bundle/ruby/2.7.0/gems/activerecord-6.1.4.1/lib/active_record/transactions.rb:290:in `transaction'
  /srv/gitlab/vendor/bundle/ruby/2.7.0/gems/activerecord-6.1.4.1/lib/active_record/locking/pessimistic.rb:86:in `with_lock'
  /srv/gitlab/app/services/terraform/remote_state_handler.rb:61:in `block in retrieve_with_lock'
  /srv/gitlab/app/services/terraform/remote_state_handler.rb:61:in `tap'
  /srv/gitlab/app/services/terraform/remote_state_handler.rb:61:in `retrieve_with_lock'
  /srv/gitlab/app/services/terraform/remote_state_handler.rb:17:in `handle_with_lock'
  /srv/gitlab/lib/api/terraform/state.rb:57:in `block (3 levels) in <class:State>'
  /srv/gitlab/vendor/bundle/ruby/2.7.0/gems/grape-1.5.2/lib/grape/endpoint.rb:59:in `call'
  /srv/gitlab/vendor/bundle/ruby/2.7.0/gems/grape-1.5.2/lib/grape/endpoint.rb:59:in `block (2 levels) in generate_api_method'
  /srv/gitlab/vendor/bundle/ruby/2.7.0/gems/activesupport-6.1.4.1/lib/active_support/notifications.rb:205:in `instrument'
  /srv/gitlab/vendor/bundle/ruby/2.7.0/gems/grape-1.5.2/lib/grape/endpoint.rb:58:in `block in generate_api_method'
  /srv/gitlab/vendor/bundle/ruby/2.7.0/gems/grape-1.5.2/lib/grape/endpoint.rb:341:in `execute'
  /srv/gitlab/vendor/bundle/ruby/2.7.0/gems/grape-1.5.2/lib/grape/endpoint.rb:267:in `block in run'
  /srv/gitlab/vendor/bundle/ruby/2.7.0/gems/activesupport-6.1.4.1/lib/active_support/notifications.rb:205:in `instrument'
  /srv/gitlab/vendor/bundle/ruby/2.7.0/gems/grape-1.5.2/lib/grape/endpoint.rb:247:in `run'
  /srv/gitlab/vendor/bundle/ruby/2.7.0/gems/grape-1.5.2/lib/grape/endpoint.rb:322:in `block in build_stack'
  /srv/gitlab/vendor/bundle/ruby/2.7.0/gems/grape-1.5.2/lib/grape/middleware/base.rb:36:in `call!'
  /srv/gitlab/vendor/bundle/ruby/2.7.0/gems/grape-1.5.2/lib/grape/middleware/base.rb:29:in `call'
  /srv/gitlab/vendor/bundle/ruby/2.7.0/gems/grape-1.5.2/lib/grape/middleware/base.rb:36:in `call!'
  /srv/gitlab/vendor/bundle/ruby/2.7.0/gems/grape-1.5.2/lib/grape/middleware/base.rb:29:in `call'
  /srv/gitlab/ee/lib/gitlab/middleware/ip_restrictor.rb:14:in `block in call'
  /srv/gitlab/ee/lib/gitlab/ip_address_state.rb:10:in `with'
  /srv/gitlab/ee/lib/gitlab/middleware/ip_restrictor.rb:13:in `call'
  /srv/gitlab/vendor/bundle/ruby/2.7.0/gems/grape-1.5.2/lib/grape/middleware/base.rb:36:in `call!'
  /srv/gitlab/vendor/bundle/ruby/2.7.0/gems/grape-1.5.2/lib/grape/middleware/base.rb:29:in `call'
  /srv/gitlab/lib/api/api_guard.rb:213:in `call'
  /srv/gitlab/vendor/bundle/ruby/2.7.0/gems/grape-1.5.2/lib/grape/middleware/base.rb:36:in `call!'
  /srv/gitlab/vendor/bundle/ruby/2.7.0/gems/grape-1.5.2/lib/grape/middleware/base.rb:29:in `call'
  /srv/gitlab/vendor/bundle/ruby/2.7.0/gems/rack-oauth2-1.16.0/lib/rack/oauth2/server/resource.rb:20:in `_call'
  /srv/gitlab/vendor/bundle/ruby/2.7.0/gems/rack-oauth2-1.16.0/lib/rack/oauth2/server/resource/bearer.rb:8:in `_call'
  /srv/gitlab/vendor/bundle/ruby/2.7.0/gems/rack-oauth2-1.16.0/lib/rack/oauth2/server/abstract/handler.rb:17:in `call'
  /srv/gitlab/vendor/bundle/ruby/2.7.0/gems/grape-1.5.2/lib/grape/middleware/error.rb:39:in `block in call!'
  /srv/gitlab/vendor/bundle/ruby/2.7.0/gems/grape-1.5.2/lib/grape/middleware/error.rb:38:in `catch'
  /srv/gitlab/vendor/bundle/ruby/2.7.0/gems/grape-1.5.2/lib/grape/middleware/error.rb:38:in `call!'
  /srv/gitlab/vendor/bundle/ruby/2.7.0/gems/grape-1.5.2/lib/grape/middleware/base.rb:29:in `call'
  /srv/gitlab/vendor/bundle/ruby/2.7.0/gems/grape_logging-1.8.3/lib/grape_logging/middleware/request_logger.rb:60:in `block in call!'
  /srv/gitlab/vendor/bundle/ruby/2.7.0/gems/grape_logging-1.8.3/lib/grape_logging/middleware/request_logger.rb:58:in `catch'
  /srv/gitlab/vendor/bundle/ruby/2.7.0/gems/grape_logging-1.8.3/lib/grape_logging/middleware/request_logger.rb:58:in `call!'
  /srv/gitlab/vendor/bundle/ruby/2.7.0/gems/grape-1.5.2/lib/grape/middleware/base.rb:29:in `call'
  /srv/gitlab/vendor/bundle/ruby/2.7.0/gems/rack-2.2.3/lib/rack/head.rb:12:in `call'
  /srv/gitlab/vendor/bundle/ruby/2.7.0/gems/grape-1.5.2/lib/grape/endpoint.rb:231:in `call!'
  /srv/gitlab/vendor/bundle/ruby/2.7.0/gems/grape-1.5.2/lib/grape/endpoint.rb:225:in `call'
  /srv/gitlab/vendor/bundle/ruby/2.7.0/gems/grape-1.5.2/lib/grape/router/route.rb:58:in `exec'
  /srv/gitlab/vendor/bundle/ruby/2.7.0/gems/grape-1.5.2/lib/grape/router.rb:116:in `process_route'
  /srv/gitlab/vendor/bundle/ruby/2.7.0/gems/grape-1.5.2/lib/grape/router.rb:72:in `block in identity'
  /srv/gitlab/vendor/bundle/ruby/2.7.0/gems/grape-1.5.2/lib/grape/router.rb:91:in `transaction'
  /srv/gitlab/vendor/bundle/ruby/2.7.0/gems/grape-1.5.2/lib/grape/router.rb:70:in `identity'
  /srv/gitlab/vendor/bundle/ruby/2.7.0/gems/grape-1.5.2/lib/grape/router.rb:55:in `block in call'
  /srv/gitlab/vendor/bundle/ruby/2.7.0/gems/grape-1.5.2/lib/grape/router.rb:132:in `with_optimization'
  /srv/gitlab/vendor/bundle/ruby/2.7.0/gems/grape-1.5.2/lib/grape/router.rb:54:in `call'
  /srv/gitlab/vendor/bundle/ruby/2.7.0/gems/grape-1.5.2/lib/grape/api/instance.rb:167:in `call'
  /srv/gitlab/vendor/bundle/ruby/2.7.0/gems/grape-1.5.2/lib/grape/api/instance.rb:71:in `call!'
  /srv/gitlab/vendor/bundle/ruby/2.7.0/gems/grape-1.5.2/lib/grape/api/instance.rb:66:in `call'
  /srv/gitlab/vendor/bundle/ruby/2.7.0/gems/actionpack-6.1.4.1/lib/action_dispatch/routing/mapper.rb:20:in `block in <class:Constraints>'
  /srv/gitlab/vendor/bundle/ruby/2.7.0/gems/actionpack-6.1.4.1/lib/action_dispatch/routing/mapper.rb:49:in `serve'
  /srv/gitlab/vendor/bundle/ruby/2.7.0/gems/actionpack-6.1.4.1/lib/action_dispatch/journey/router.rb:50:in `block in serve'
  /srv/gitlab/vendor/bundle/ruby/2.7.0/gems/actionpack-6.1.4.1/lib/action_dispatch/journey/router.rb:32:in `each'
  /srv/gitlab/vendor/bundle/ruby/2.7.0/gems/actionpack-6.1.4.1/lib/action_dispatch/journey/router.rb:32:in `serve'
  /srv/gitlab/vendor/bundle/ruby/2.7.0/gems/actionpack-6.1.4.1/lib/action_dispatch/routing/route_set.rb:842:in `call'
  /srv/gitlab/vendor/bundle/ruby/2.7.0/gems/gitlab-experiment-0.6.4/lib/gitlab/experiment/middleware.rb:19:in `call'
  /srv/gitlab/vendor/bundle/ruby/2.7.0/gems/omniauth-1.9.1/lib/omniauth/strategy.rb:192:in `call!'
  /srv/gitlab/vendor/bundle/ruby/2.7.0/gems/omniauth-1.9.1/lib/omniauth/strategy.rb:169:in `call'
  /srv/gitlab/vendor/bundle/ruby/2.7.0/gems/omniauth-1.9.1/lib/omniauth/strategy.rb:192:in `call!'
  /srv/gitlab/vendor/bundle/ruby/2.7.0/gems/omniauth-1.9.1/lib/omniauth/strategy.rb:169:in `call'
  /srv/gitlab/vendor/bundle/ruby/2.7.0/gems/omniauth-1.9.1/lib/omniauth/strategy.rb:420:in `call_app!'
  /srv/gitlab/vendor/bundle/ruby/2.7.0/gems/omniauth-saml-1.10.0/lib/omniauth/strategies/saml.rb:86:in `other_phase'
  /srv/gitlab/vendor/bundle/ruby/2.7.0/gems/omniauth-1.9.1/lib/omniauth/strategy.rb:190:in `call!'
  /srv/gitlab/vendor/bundle/ruby/2.7.0/gems/omniauth-1.9.1/lib/omniauth/strategy.rb:169:in `call'
  /srv/gitlab/vendor/bundle/ruby/2.7.0/gems/flipper-0.21.0/lib/flipper/middleware/memoizer.rb:77:in `memoized_call'
  /srv/gitlab/vendor/bundle/ruby/2.7.0/gems/flipper-0.21.0/lib/flipper/middleware/memoizer.rb:42:in `call'
  /srv/gitlab/lib/gitlab/metrics/elasticsearch_rack_middleware.rb:16:in `call'
  /srv/gitlab/lib/gitlab/middleware/rails_queue_duration.rb:33:in `call'
  /srv/gitlab/lib/gitlab/middleware/speedscope.rb:13:in `call'
  /srv/gitlab/lib/gitlab/request_profiler/middleware.rb:17:in `call'
  /srv/gitlab/lib/gitlab/database/load_balancing/rack_middleware.rb:23:in `call'
  /srv/gitlab/lib/gitlab/metrics/rack_middleware.rb:16:in `block in call'
  /srv/gitlab/lib/gitlab/metrics/web_transaction.rb:21:in `run'
  /srv/gitlab/lib/gitlab/metrics/rack_middleware.rb:16:in `call'
  /srv/gitlab/lib/gitlab/jira/middleware.rb:19:in `call'
  /srv/gitlab/lib/gitlab/middleware/go.rb:20:in `call'
  /srv/gitlab/lib/gitlab/etag_caching/middleware.rb:21:in `call'
  /srv/gitlab/vendor/bundle/ruby/2.7.0/gems/batch-loader-2.0.1/lib/batch_loader/middleware.rb:11:in `call'
  /srv/gitlab/vendor/bundle/ruby/2.7.0/gems/rack-attack-6.3.0/lib/rack/attack.rb:97:in `call'
  /srv/gitlab/vendor/bundle/ruby/2.7.0/gems/apollo_upload_server-2.1.0/lib/apollo_upload_server/middleware.rb:19:in `call'
  /srv/gitlab/lib/gitlab/middleware/multipart.rb:173:in `call'
  /srv/gitlab/vendor/bundle/ruby/2.7.0/gems/rack-attack-6.3.0/lib/rack/attack.rb:111:in `call'
  /srv/gitlab/vendor/bundle/ruby/2.7.0/gems/warden-1.2.8/lib/warden/manager.rb:36:in `block in call'
  /srv/gitlab/vendor/bundle/ruby/2.7.0/gems/warden-1.2.8/lib/warden/manager.rb:34:in `catch'
  /srv/gitlab/vendor/bundle/ruby/2.7.0/gems/warden-1.2.8/lib/warden/manager.rb:34:in `call'
  /srv/gitlab/vendor/bundle/ruby/2.7.0/gems/rack-cors-1.0.6/lib/rack/cors.rb:98:in `call'
  /srv/gitlab/vendor/bundle/ruby/2.7.0/gems/rack-2.2.3/lib/rack/tempfile_reaper.rb:15:in `call'
  /srv/gitlab/vendor/bundle/ruby/2.7.0/gems/rack-2.2.3/lib/rack/etag.rb:27:in `call'
  /srv/gitlab/vendor/bundle/ruby/2.7.0/gems/rack-2.2.3/lib/rack/conditional_get.rb:40:in `call'
  /srv/gitlab/vendor/bundle/ruby/2.7.0/gems/rack-2.2.3/lib/rack/head.rb:12:in `call'
  /srv/gitlab/vendor/bundle/ruby/2.7.0/gems/actionpack-6.1.4.1/lib/action_dispatch/http/permissions_policy.rb:22:in `call'
  /srv/gitlab/vendor/bundle/ruby/2.7.0/gems/actionpack-6.1.4.1/lib/action_dispatch/http/content_security_policy.rb:18:in `call'
  /srv/gitlab/lib/gitlab/middleware/read_only/controller.rb:50:in `call'
  /srv/gitlab/lib/gitlab/middleware/read_only.rb:18:in `call'
  /srv/gitlab/vendor/bundle/ruby/2.7.0/gems/rack-2.2.3/lib/rack/session/abstract/id.rb:266:in `context'
  /srv/gitlab/vendor/bundle/ruby/2.7.0/gems/rack-2.2.3/lib/rack/session/abstract/id.rb:260:in `call'
  /srv/gitlab/vendor/bundle/ruby/2.7.0/gems/actionpack-6.1.4.1/lib/action_dispatch/middleware/cookies.rb:689:in `call'
  /srv/gitlab/lib/gitlab/middleware/same_site_cookies.rb:27:in `call'
  /srv/gitlab/vendor/bundle/ruby/2.7.0/gems/actionpack-6.1.4.1/lib/action_dispatch/middleware/callbacks.rb:27:in `block in call'
  /srv/gitlab/vendor/bundle/ruby/2.7.0/gems/activesupport-6.1.4.1/lib/active_support/callbacks.rb:98:in `run_callbacks'
  /srv/gitlab/vendor/bundle/ruby/2.7.0/gems/actionpack-6.1.4.1/lib/action_dispatch/middleware/callbacks.rb:26:in `call'
  /srv/gitlab/lib/gitlab/middleware/handle_malformed_strings.rb:21:in `call'
  /srv/gitlab/vendor/bundle/ruby/2.7.0/gems/actionpack-6.1.4.1/lib/action_dispatch/middleware/actionable_exceptions.rb:18:in `call'
  /srv/gitlab/vendor/bundle/ruby/2.7.0/gems/actionpack-6.1.4.1/lib/action_dispatch/middleware/debug_exceptions.rb:29:in `call'
  /srv/gitlab/vendor/bundle/ruby/2.7.0/gems/actionpack-6.1.4.1/lib/action_dispatch/middleware/show_exceptions.rb:33:in `call'
  /srv/gitlab/lib/gitlab/middleware/basic_health_check.rb:25:in `call'
  /srv/gitlab/vendor/bundle/ruby/2.7.0/gems/railties-6.1.4.1/lib/rails/rack/logger.rb:37:in `call_app'
  /srv/gitlab/vendor/bundle/ruby/2.7.0/gems/railties-6.1.4.1/lib/rails/rack/logger.rb:26:in `block in call'
  /srv/gitlab/vendor/bundle/ruby/2.7.0/gems/activesupport-6.1.4.1/lib/active_support/tagged_logging.rb:99:in `block in tagged'
  /srv/gitlab/vendor/bundle/ruby/2.7.0/gems/activesupport-6.1.4.1/lib/active_support/tagged_logging.rb:37:in `tagged'
  /srv/gitlab/vendor/bundle/ruby/2.7.0/gems/activesupport-6.1.4.1/lib/active_support/tagged_logging.rb:99:in `tagged'
  /srv/gitlab/vendor/bundle/ruby/2.7.0/gems/railties-6.1.4.1/lib/rails/rack/logger.rb:26:in `call'
  /srv/gitlab/vendor/bundle/ruby/2.7.0/gems/actionpack-6.1.4.1/lib/action_dispatch/middleware/remote_ip.rb:81:in `call'
  /srv/gitlab/lib/gitlab/middleware/handle_ip_spoof_attack_error.rb:25:in `call'
  /srv/gitlab/lib/gitlab/middleware/request_context.rb:21:in `call'
  /srv/gitlab/vendor/bundle/ruby/2.7.0/gems/request_store-1.5.0/lib/request_store/middleware.rb:19:in `call'
  /srv/gitlab/vendor/bundle/ruby/2.7.0/gems/rack-2.2.3/lib/rack/method_override.rb:24:in `call'
  /srv/gitlab/vendor/bundle/ruby/2.7.0/gems/rack-2.2.3/lib/rack/runtime.rb:22:in `call'
  /srv/gitlab/vendor/bundle/ruby/2.7.0/gems/rack-timeout-0.5.2/lib/rack/timeout/core.rb:123:in `block in call'
  /srv/gitlab/vendor/bundle/ruby/2.7.0/gems/rack-timeout-0.5.2/lib/rack/timeout/support/timeout.rb:19:in `timeout'
  /srv/gitlab/vendor/bundle/ruby/2.7.0/gems/rack-timeout-0.5.2/lib/rack/timeout/core.rb:122:in `call'
  /srv/gitlab/config/initializers/fix_local_cache_middleware.rb:11:in `call'
  /srv/gitlab/vendor/bundle/ruby/2.7.0/gems/actionpack-6.1.4.1/lib/action_dispatch/middleware/executor.rb:14:in `call'
  /srv/gitlab/lib/gitlab/middleware/rack_multipart_tempfile_factory.rb:19:in `call'
  /srv/gitlab/vendor/bundle/ruby/2.7.0/gems/rack-2.2.3/lib/rack/sendfile.rb:110:in `call'
  /srv/gitlab/lib/gitlab/middleware/sidekiq_web_static.rb:20:in `call'
  /srv/gitlab/vendor/bundle/ruby/2.7.0/gems/actionpack-6.1.4.1/lib/action_dispatch/middleware/host_authorization.rb:92:in `call'
  /srv/gitlab/lib/gitlab/metrics/requests_rack_middleware.rb:75:in `call'
  /srv/gitlab/vendor/bundle/ruby/2.7.0/gems/gitlab-labkit-0.21.1/lib/labkit/middleware/rack.rb:19:in `block in call'
  /srv/gitlab/vendor/bundle/ruby/2.7.0/gems/gitlab-labkit-0.21.1/lib/labkit/context.rb:39:in `with_context'
  /srv/gitlab/vendor/bundle/ruby/2.7.0/gems/gitlab-labkit-0.21.1/lib/labkit/middleware/rack.rb:18:in `call'
  /srv/gitlab/vendor/bundle/ruby/2.7.0/gems/actionpack-6.1.4.1/lib/action_dispatch/middleware/request_id.rb:26:in `call'
  /srv/gitlab/vendor/bundle/ruby/2.7.0/gems/sentry-raven-3.1.2/lib/raven/integrations/rack.rb:51:in `call'
  /srv/gitlab/vendor/bundle/ruby/2.7.0/gems/railties-6.1.4.1/lib/rails/engine.rb:539:in `call'
  /srv/gitlab/vendor/bundle/ruby/2.7.0/gems/railties-6.1.4.1/lib/rails/railtie.rb:207:in `public_send'
  /srv/gitlab/vendor/bundle/ruby/2.7.0/gems/railties-6.1.4.1/lib/rails/railtie.rb:207:in `method_missing'
  /srv/gitlab/lib/gitlab/middleware/release_env.rb:12:in `call'
  /srv/gitlab/vendor/bundle/ruby/2.7.0/gems/rack-2.2.3/lib/rack/urlmap.rb:74:in `block in call'
  /srv/gitlab/vendor/bundle/ruby/2.7.0/gems/rack-2.2.3/lib/rack/urlmap.rb:58:in `each'
  /srv/gitlab/vendor/bundle/ruby/2.7.0/gems/rack-2.2.3/lib/rack/urlmap.rb:58:in `call'
  /srv/gitlab/vendor/bundle/ruby/2.7.0/gems/puma-5.3.2/lib/puma/configuration.rb:249:in `call'
  /srv/gitlab/vendor/bundle/ruby/2.7.0/gems/puma-5.3.2/lib/puma/request.rb:77:in `block in handle_request'
  /srv/gitlab/vendor/bundle/ruby/2.7.0/gems/puma-5.3.2/lib/puma/thread_pool.rb:338:in `with_force_shutdown'
  /srv/gitlab/vendor/bundle/ruby/2.7.0/gems/puma-5.3.2/lib/puma/request.rb:76:in `handle_request'
  /srv/gitlab/vendor/bundle/ruby/2.7.0/gems/puma-5.3.2/lib/puma/server.rb:438:in `process_client'
  /srv/gitlab/vendor/bundle/ruby/2.7.0/gems/puma-5.3.2/lib/puma/thread_pool.rb:145:in `block in spawn_thread'
  Parameters: {"model"=>"project", "mounted_as"=>"avatar", "id"=>"2", "filename"=>"sharkmob-red.png"}
*** /var/log/gitlab/api_json.log ***
{"time":"2021-11-02T17:07:21.927Z","severity":"INFO","duration_s":0.07986,"db_duration_s":0.02038,"view_duration_s":0.05948,"status":500,"method":"POST","path":"/api/v4/projects/18/terraform/state/dev-init","params":[{"key":"ID","value":"ef14a2c0-d388-064a-2a21-bb667713229e"},{"key":"version","value":4},{"key":"terraform_version","value":"1.0.8"},{"key":"serial","value":0},{"key":"lineage","value":"d8586bde-cad3-e6e9-aaab-787fef060a54"},{"key":"outputs","value":{}},{"key":"resources","value":[{"module":"module.init","mode":"managed","type":"google_dns_record_set","name":"ns_records","provider":"provider[\"registry.terraform.io/hashicorp/google\"].overwatch_default","instances":[]},{"module":"module.init","mode":"managed","type":"google_project_iam_member","name":"project","provider":"provider[\"registry.terraform.io/hashicorp/google\"].project_default","instances":[]},{"module":"module.init","mode":"managed","type":"google_service_account_key","name":"tf","provider":"provider[\"registry.terraform.io/hashicorp/google\"].project_default","instances":[]},{"module":"module.init","mode":"managed","type":"null_resource","name":"wait_for_compute_api","provider":"provider[\"registry.terraform.io/hashicorp/null\"]","instances":[{"schema_version":0,"attributes":{"id":"8824741656771903754","triggers":null},"sensitive_attributes":[],"private":"bnVsbA==","dependencies":["module.init.module.project_services.google_project_iam_member.project_service_identity_roles","module.init.module.project_services.google_project_service.project_services","module.init.module.project_services.google_project_service_identity.project_service_identities"]}]},{"module":"module.init.module.project_services","mode":"managed","type":"google_project_service","name":"project_services","provider":"provider[\"registry.terraform.io/hashicorp/google\"].project_default","instances":[{"index_key":"[FILTERED]","schema_version":0,"attributes":{"disable_dependent_services":null,"disable_on_destroy":null,"id":"sharkmob-p2-dev/cloudresourcemanager.googleapis.com","project":"sharkmob-p2-dev","service":"cloudresourcemanager.googleapis.com","timeouts":null},"sensitive_attributes":[],"private":"eyJlMmJmYjczMC1lY2FhLTExZTYtOGY4OC0zNDM2M2JjN2M0YzAiOnsiY3JlYXRlIjoxMjAwMDAwMDAwMDAwLCJkZWxldGUiOjEyMDAwMDAwMDAwMDAsInJlYWQiOjYwMDAwMDAwMDAwMCwidXBkYXRlIjoxMjAwMDAwMDAwMDAwfX0="},{"index_key":"[FILTERED]","schema_version":0,"attributes":{"disable_dependent_services":null,"disable_on_destroy":null,"id":"sharkmob-p2-dev/compute.googleapis.com","project":"sharkmob-p2-dev","service":"compute.googleapis.com","timeouts":null},"sensitive_attributes":[],"private":"eyJlMmJmYjczMC1lY2FhLTExZTYtOGY4OC0zNDM2M2JjN2M0YzAiOnsiY3JlYXRlIjoxMjAwMDAwMDAwMDAwLCJkZWxldGUiOjEyMDAwMDAwMDAwMDAsInJlYWQiOjYwMDAwMDAwMDAwMCwidXBkYXRlIjoxMjAwMDAwMDAwMDAwfX0="},{"index_key":"[FILTERED]","schema_version":0,"attributes":{"disable_dependent_services":null,"disable_on_destroy":null,"id":"sharkmob-p2-dev/container.googleapis.com","project":"sharkmob-p2-dev","service":"container.googleapis.com","timeouts":null},"sensitive_attributes":[],"private":"eyJlMmJmYjczMC1lY2FhLTExZTYtOGY4OC0zNDM2M2JjN2M0YzAiOnsiY3JlYXRlIjoxMjAwMDAwMDAwMDAwLCJkZWxldGUiOjEyMDAwMDAwMDAwMDAsInJlYWQiOjYwMDAwMDAwMDAwMCwidXBkYXRlIjoxMjAwMDAwMDAwMDAwfX0="},{"index_key":"[FILTERED]","schema_version":0,"attributes":{"disable_dependent_services":null,"disable_on_destroy":null,"id":"sharkmob-p2-dev/dns.googleapis.com","project":"sharkmob-p2-dev","service":"dns.googleapis.com","timeouts":null},"sensitive_attributes":[],"private":"eyJlMmJmYjczMC1lY2FhLTExZTYtOGY4OC0zNDM2M2JjN2M0YzAiOnsiY3JlYXRlIjoxMjAwMDAwMDAwMDAwLCJkZWxldGUiOjEyMDAwMDAwMDAwMDAsInJlYWQiOjYwMDAwMDAwMDAwMCwidXBkYXRlIjoxMjAwMDAwMDAwMDAwfX0="},{"index_key":"[FILTERED]","schema_version":0,"attributes":{"disable_dependent_services":null,"disable_on_destroy":null,"id":"sharkmob-p2-dev/iam.googleapis.com","project":"sharkmob-p2-dev","service":"iam.googleapis.com","timeouts":null},"sensitive_attributes":[],"private":"eyJlMmJmYjczMC1lY2FhLTExZTYtOGY4OC0zNDM2M2JjN2M0YzAiOnsiY3JlYXRlIjoxMjAwMDAwMDAwMDAwLCJkZWxldGUiOjEyMDAwMDAwMDAwMDAsInJlYWQiOjYwMDAwMDAwMDAwMCwidXBkYXRlIjoxMjAwMDAwMDAwMDAwfX0="},{"index_key":"[FILTERED]","schema_version":0,"attributes":{"disable_dependent_services":null,"disable_on_destroy":null,"id":"sharkmob-p2-dev/iap.googleapis.com","project":"sharkmob-p2-dev","service":"iap.googleapis.com","timeouts":null},"sensitive_attributes":[],"private":"eyJlMmJmYjczMC1lY2FhLTExZTYtOGY4OC0zNDM2M2JjN2M0YzAiOnsiY3JlYXRlIjoxMjAwMDAwMDAwMDAwLCJkZWxldGUiOjEyMDAwMDAwMDAwMDAsInJlYWQiOjYwMDAwMDAwMDAwMCwidXBkYXRlIjoxMjAwMDAwMDAwMDAwfX0="},{"index_key":"[FILTERED]","schema_version":0,"attributes":{"disable_dependent_services":null,"disable_on_destroy":null,"id":"sharkmob-p2-dev/monitoring.googleapis.com","project":"sharkmob-p2-dev","service":"monitoring.googleapis.com","timeouts":null},"sensitive_attributes":[],"private":"eyJlMmJmYjczMC1lY2FhLTExZTYtOGY4OC0zNDM2M2JjN2M0YzAiOnsiY3JlYXRlIjoxMjAwMDAwMDAwMDAwLCJkZWxldGUiOjEyMDAwMDAwMDAwMDAsInJlYWQiOjYwMDAwMDAwMDAwMCwidXBkYXRlIjoxMjAwMDAwMDAwMDAwfX0="},{"index_key":"[FILTERED]","schema_version":0,"attributes":{"disable_dependent_services":null,"disable_on_destroy":null,"id":"sharkmob-p2-dev/redis.googleapis.com","project":"sharkmob-p2-dev","service":"redis.googleapis.com","timeouts":null},"sensitive_attributes":[],"private":"eyJlMmJmYjczMC1lY2FhLTExZTYtOGY4OC0zNDM2M2JjN2M0YzAiOnsiY3JlYXRlIjoxMjAwMDAwMDAwMDAwLCJkZWxldGUiOjEyMDAwMDAwMDAwMDAsInJlYWQiOjYwMDAwMDAwMDAwMCwidXBkYXRlIjoxMjAwMDAwMDAwMDAwfX0="},{"index_key":"[FILTERED]","schema_version":0,"attributes":{"disable_dependent_services":null,"disable_on_destroy":null,"id":"sharkmob-p2-dev/servicenetworking.googleapis.com","project":"sharkmob-p2-dev","service":"servicenetworking.googleapis.com","timeouts":null},"sensitive_attributes":[],"private":"eyJlMmJmYjczMC1lY2FhLTExZTYtOGY4OC0zNDM2M2JjN2M0YzAiOnsiY3JlYXRlIjoxMjAwMDAwMDAwMDAwLCJkZWxldGUiOjEyMDAwMDAwMDAwMDAsInJlYWQiOjYwMDAwMDAwMDAwMCwidXBkYXRlIjoxMjAwMDAwMDAwMDAwfX0="},{"index_key":"[FILTERED]","schema_version":0,"attributes":{"disable_dependent_services":null,"disable_on_destroy":null,"id":"sharkmob-p2-dev/sqladmin.googleapis.com","project":"sharkmob-p2-dev","service":"sqladmin.googleapis.com","timeouts":null},"sensitive_attributes":[],"private":"eyJlMmJmYjczMC1lY2FhLTExZTYtOGY4OC0zNDM2M2JjN2M0YzAiOnsiY3JlYXRlIjoxMjAwMDAwMDAwMDAwLCJkZWxldGUiOjEyMDAwMDAwMDAwMDAsInJlYWQiOjYwMDAwMDAwMDAwMCwidXBkYXRlIjoxMjAwMDAwMDAwMDAwfX0="}]}]}],"host":"gitlab.sharkmob.cloud","remote_ip":"193.12.14.186, 193.12.14.186","ua":"Go-http-client/1.1","route":"/api/:version/projects/:id/terraform/state/:name","user_id":2,"username":"jmh","exception.class":"ActiveRecord::RecordNotUnique","exception.message":"PG::UniqueViolation: ERROR:  duplicate key value violates unique constraint \"index_terraform_state_versions_on_state_id_and_version\"\nDETAIL:  Key (terraform_state_id, version)=(1, 0) already exists.\n","exception.backtrace":["lib/gitlab/database/load_balancing/connection_proxy.rb:119:in `block in write_using_load_balancer'","lib/gitlab/database/load_balancing/load_balancer.rb:107:in `block in read_write'","lib/gitlab/database/load_balancing/load_balancer.rb:167:in `retry_with_backoff'","lib/gitlab/database/load_balancing/load_balancer.rb:105:in `read_write'","lib/gitlab/database/load_balancing/connection_proxy.rb:118:in `write_using_load_balancer'","lib/gitlab/database/load_balancing/connection_proxy.rb:60:in `block (2 levels) in <class:ConnectionProxy>'","lib/gitlab/database/load_balancing/connection_proxy.rb:119:in `block in write_using_load_balancer'","lib/gitlab/database/load_balancing/load_balancer.rb:107:in `block in read_write'","lib/gitlab/database/load_balancing/load_balancer.rb:167:in `retry_with_backoff'","lib/gitlab/database/load_balancing/load_balancer.rb:105:in `read_write'","lib/gitlab/database/load_balancing/connection_proxy.rb:118:in `write_using_load_balancer'","lib/gitlab/database/load_balancing/connection_proxy.rb:70:in `transaction'","app/models/terraform/state.rb:90:in `create_new_version!'","app/models/terraform/state.rb:49:in `update_file!'","lib/api/terraform/state.rb:58:in `block (4 levels) in <class:State>'","app/services/terraform/remote_state_handler.rb:20:in `block in handle_with_lock'","app/services/terraform/remote_state_handler.rb:61:in `block (2 levels) in retrieve_with_lock'","lib/gitlab/database/load_balancing/connection_proxy.rb:119:in `block in write_using_load_balancer'","lib/gitlab/daid":2,"username":"jmh","ua":"Mozilla
tabase/load_balancing/load_balancer.rb:107:in `block in read_write'","lib/gitlab/database/load_balancing/load_balancer.rb:167:in `retry_with_backoff'","lib/gitlab/database/load_balancing/load_balancer.rb:105:in `read_write'","lib/gitlab/database/load_balancing/connection_proxy.rb:118:in `write_using_load_balancer'","lib/gitlab/database/load_balancing/connection_proxy.rb:70:in `transaction'","lib/gitlab/database.rb:271:in `block in transaction'","lib/gitlab/database.rb:270:in `transaction'","app/services/terraform/remote_state_handler.rb:61:in `block in retrieve_with_lock'","app/services/terraform/remote_state_handler.rb:61:in `tap'","app/services/terraform/remote_state_handler.rb:61:in `retrieve_with_lock'","app/services/terraform/remote_state_handler.rb:17:in `handle_with_lock'","lib/api/terraform/state.rb:57:in `block (3 levels) in <class:State>'","ee/lib/gitlab/middleware/ip_restrictor.rb:14:in `block in call'","ee/lib/gitlab/ip_address_state.rb:10:in `with'","ee/lib/gitlab/middleware/ip_restrictor.rb:13:in `call'","lib/api/api_guard.rb:213:in `call'","lib/gitlab/metrics/elasticsearch_rack_middleware.rb:16:in `call'","lib/gitlab/middleware/rails_queue_duration.rb:33:in `call'","lib/gitlab/middleware/speedscope.rb:13:in `call'","lib/gitlab/request_profiler/middleware.rb:17:in `call'","lib/gitlab/database/load_balancing/rack_middleware.rb:23:in `call'","lib/gitlab/metrics/rack_middleware.rb:16:in `block in call'","lib/gitlab/metrics/web_transaction.rb:21:in `run'","lib/gitlab/metrics/rack_middleware.rb:16:in `call'","lib/gitlab/jira/middleware.rb:19:in `call'","lib/gitlab/middleware/go.rb:20:in `call'","lib/gitlab/etag_caching/middleware.rb:21:in `call'","lib/gitlab/middleware/multipart.rb:173:in `call'","lib/gitlab/middleware/read_only/controller.rb:50:in `call'","lib/gitlab/middleware/read_only.rb:18:in `call'","lib/gitlab/middleware/same_site_cookies.rb:27:in `call'","lib/gitlab/middleware/handle_malformed_strings.rb:21:in `call'","lib/gitlab/middleware/basic_health_check.rb:25:in `call'","lib/gitlab/middleware/handle_ip_spoof_attack_error.rb:25:in `call'","lib/gitlab/middleware/request_context.rb:21:in `call'","config/initializers/fix_local_cache_middleware.rb:11:in `call'","lib/gitlab/middleware/rack_multipart_tempfile_factory.rb:19:in `call'","lib/gitlab/middleware/sidekiq_web_static.rb:20:in `call'","lib/gitlab/metrics/requests_rack_middleware.rb:75:in `call'","lib/gitlab/middleware/release_env.rb:12:in `call'"],"queue_duration_s":0.011441,"redis_calls":2,"redis_duration_s":0.001082,"redis_write_bytes":101,"redis_shared_state_calls":2,"redis_shared_state_duration_s":0.001082,"redis_shared_state_write_bytes":101,"db_count":21,"db_write_count":6,"db_cached_count":0,"db_replica_count":0,"db_replica_cached_count":0,"db_replica_wal_count":0,"db_replica_wal_cached_count":0,"db_primary_count":21,"db_primary_cached_count":0,"db_primary_wal_count":0,"db_primary_wal_cached_count":0,"db_replica_duration_s":0.0,"db_primary_duration_s":0.021,"cpu_s":0.073534,"mem_objects":21934,"mem_bytes":2022536,"mem_mallocs":6167,"mem_total_bytes":2899896,"pid":31,"correlation_id":"01FKGSX8394ZNAB4GSDRRZ9AG8","meta.project":"p2/tf/gcp-core","meta.root_namespace":"p2","meta.caller_id":"POST /api/:version/projects/:id/terraform/state/:name","meta.remote_ip":"193.12.14.186","meta.feature_category":"infrastructure_as_code","meta.client_id":"ip/","content_length":"8519"}

*** /var/log/gitlab/exceptions_json.log ***
{"severity":"ERROR","time":"2021-11-02T17:07:21.924Z","correlation_id":"01FKGSX8394ZNAB4GSDRRZ9AG8","exception.class":"ActiveRecord::RecordNotUnique","exception.message":"PG::UniqueViolation: ERROR:  duplicate key value violates unique constraint \"index_terraform_state_versions_on_state_id_and_version\"\nDETAIL:  Key (terraform_state_id, version)=(1, 0) already exists.\n","exception.backtrace":["lib/gitlab/database/load_balancing/connection_proxy.rb:119:in `block in write_using_load_balancer'","lib/gitlab/database/load_balancing/load_balancer.rb:107:in `block in read_write'","lib/gitlab/database/load_balancing/load_balancer.rb:167:in `retry_with_backoff'","lib/gitlab/database/load_balancing/load_balancer.rb:105:in `read_write'","lib/gitlab/database/load_balancing/connection_proxy.rb:118:in `write_using_load_balancer'","lib/gitlab/database/load_balancing/connection_proxy.rb:60:in `block (2 levels) in \u003cclass:ConnectionProxy\u003e'","lib/gitlab/database/load_balancing/connection_proxy.rb:119:in `block in write_using_load_balancer'","lib/gitlab/database/load_balancing/load_balancer.rb:107:in `block in read_write'","lib/gitlab/database/load_balancing/load_balancer.rb:167:in `retry_with_backoff'","lib/gitlab/database/load_balancing/load_balancer.rb:105:in `read_write'","lib/gitlab/database/load_balancing/connection_proxy.rb:118:in `write_using_load_balancer'","lib/gitlab/database/load_balancing/connection_proxy.rb:70:in `transaction'","app/models/terraform/state.rb:90:in `create_new_version!'","app/models/terraform/state.rb:49:in `update_file!'","lib/api/terraform/state.rb:58:in `block (4 levels) in \u003cclass:State\u003e'","app/services/terraform/remote_state_handler.rb:20:in `block in handle_with_lock'","app/services/terraform/remote_state_handler.rb:61:in `block (2 levels) in retrieve_with_lock'","lib/gitlab/database/load_balancing/connection_proxy.rb:119:in `block in write_using_load_balancer'","lib/gitlab/database/load_balancing/load_balancer.rb:107:in `block in read_write'","lib/gitlab/database/load_balancing/load_balancer.rb:167:in `retry_with_backoff'","lib/gitlab/database/load_balancing/load_balancer.rb:105:in `read_write'","lib/gitlab/database/load_balancing/connection_proxy.rb:118:in `write_using_load_balancer'","lib/gitlab/database/load_balancing/connection_proxy.rb:70:in `transaction'","lib/gitlab/database.rb:271:in `block in transaction'","lib/gitlab/database.rb:270:in `transaction'","app/services/terraform/remote_state_handler.rb:61:in `block in retrieve_with_lock'","app/services/terraform/remote_state_handler.rb:61:in `tap'","app/services/terraform/remote_state_handler.rb:61:in `retrieve_with_lock'","app/services/terraform/remote_state_handler.rb:17:in `handle_with_lock'","lib/api/terraform/state.rb:57:in `block (3 levels) in \u003cclass:State\u003e'","ee/lib/gitlab/middleware/ip_restrictor.rb:14:in `block in call'","ee/lib/gitlab/ip_address_state.rb:10:in `with'","ee/lib/gitlab/middleware/ip_restrictor.rb:13:in `call'","lib/api/api_guard.rb:213:in `call'","lib/gitlab/metrics/elasticsearch_rack_middleware.rb:16:in `call'","lib/gitlab/middleware/rails_queue_duration.rb:33:in `call'","lib/gitlab/middleware/speedscope.rb:13:in `call'","lib/gitlab/request_profiler/middleware.rb:17:in `call'","lib/gitlab/database/load_balancing/rack_middleware.rb:23:in `call'","lib/gitlab/metrics/rack_middleware.rb:16:in `block in call'","lib/gitlab/metrics/web_transaction.rb:21:in `run'","lib/gitlab/metrics/rack_middleware.rb:16:in `call'","lib/gitlab/jira/middleware.rb:19:in `call'","lib/gitlab/middleware/go.rb:20:in `call'","lib/gitlab/etag_caching/middleware.rb:21:in `call'","lib/gitlab/middleware/multipart.rb:173:in `call'","lib/gitlab/middleware/read_only/controller.rb:50:in `call'","lib/gitlab/middleware/read_only.rb:18:in `call'","lib/gitlab/middleware/same_site_cookies.rb:27:in `call'","lib/gitlab/middleware/handle_malformed_strings.rb:21:in `call'","lib/gitlab/middleware/basic_health_check.rb:25:in `call'","lib/gitlab/middleware/handle_ip_spoof_attack_error.rb:25:in `call'","lib/gitlab/middleware/request_context.rb:21:in `call'","config/initializers/fix_local_cache_middleware.rb:11:in `call'","lib/gitlab/middleware/rack_multipart_tempfile_factory.rb:19:in `call'","lib/gitlab/middleware/sidekiq_web_static.rb:20:in `call'","lib/gitlab/metrics/requests_rack_middleware.rb:75:in `call'","lib/gitlab/middleware/release_env.rb:12:in `call'"],"user.username":null,"tags.program":"web","tags.locale":"en","tags.feature_category":"infrastructure_as_code","tags.correlation_id":"01FKGSX8394ZNAB4GSDRRZ9AG8"}

Output of checks

Results of GitLab environment info

Expand for output related to GitLab environment info 
kubectl exec (kubectl get pods -lrelease=gitlab,app=task-runner -o jsonpath="{.items[].metadata.name}") -it -- gitlab-rake gitlab:env:info SANITIZE=true
Defaulted container "task-runner" out of: task-runner, certificates (init), configure (init)

System information
System:		
Proxy:		no
Current User:	git
Using RVM:	no
Ruby Version:	2.7.4p191
Gem Version:	3.2.19
Bundler Version:2.2.19
Rake Version:	13.0.6
Redis Version:	unknown
Git Version:	unknown
Sidekiq Version:6.2.2
Go Version:	unknown

GitLab information
Version:	14.4.1-ee
Revision:	abc23a14bac
Directory:	/srv/gitlab
DB Adapter:	PostgreSQL
DB Version:	13.4
URL:		https://gitlab.sharkmob.cloud
HTTP Clone URL:	https://gitlab.sharkmob.cloud/some-group/some-project.git
SSH Clone URL:	git@gitlab.sharkmob.cloud:some-group/some-project.git
Elasticsearch:	no
Geo:		no
Using LDAP:	no
Using Omniauth:	yes
Omniauth Providers: saml, google_oauth2, github

GitLab Shell
Version:	13.21.1
Repository storage paths:
- default: 	/var/opt/gitlab/repo
GitLab Shell path:		/home/git/gitlab-shell
Git:		/usr/bin/git

Results of GitLab application Check

Expand for output of GitLab Check 
Gitlab Application check is not working with the HELM deployment, results look incorrect but this is because the task-runner does not run the services it's checking for.

kubectl exec (kubectl get pods -lrelease=gitlab,app=task-runner -o jsonpath="{.items[].metadata.name}") -it -- gitlab-rake gitlab:check SANITIZE=true
Defaulted container "task-runner" out of: task-runner, certificates (init), configure (init)
Checking GitLab subtasks ...

Checking GitLab Shell ...

GitLab Shell: ... GitLab Shell version >= 13.21.1 ? ... OK (13.21.1)
Running /home/git/gitlab-shell/bin/check
gitlab-shell self-check failed
  Try fixing it:
  Make sure GitLab is running;
  Check the gitlab-shell configuration file:
  sudo -u git -H editor /home/git/gitlab-shell/config.yml
  Please fix the error above and rerun the checks.

Checking GitLab Shell ... Finished

Checking Gitaly ...

Gitaly: ... default ... OK

Checking Gitaly ... Finished

Checking Sidekiq ...

Sidekiq: ... Running? ... no
  Try fixing it:
  sudo -u git -H RAILS_ENV=production bin/background_jobs start
  For more information see:
  doc/install/installation.md in section "Install Init Script"
  see log/sidekiq.log for possible errors
  Please fix the error above and rerun the checks.

Checking Sidekiq ... Finished

Checking Incoming Email ...

Incoming Email: ... Reply by email is disabled in config/gitlab.yml

Checking Incoming Email ... Finished

Checking LDAP ...

LDAP: ... LDAP is disabled in config/gitlab.yml

Checking LDAP ... Finished

Checking GitLab App ...

Git configured correctly? ... no
Trying to fix error automatically. ...Failed
  Try fixing it:
  sudo -u git -H "/usr/bin/git" config --global core.autocrlf "input"
  For more information see:
  doc/install/installation.md in section "GitLab"
Database config exists? ... yes
All migrations up? ... yes
Database contains orphaned GroupMembers? ... no
GitLab config exists? ... yes
GitLab config up to date? ... yes
Log directory writable? ... yes
Tmp directory writable? ... yes
Uploads directory exists? ... yes
Uploads directory has correct permissions? ... yes
Uploads directory tmp has correct permissions? ... skipped (no tmp uploads folder yet)
Init script exists? ... no
  Try fixing it:
  Install the init script
  For more information see:
  doc/install/installation.md in section "Install Init Script"
  Please fix the error above and rerun the checks.
Init script up-to-date? ... can't check because of previous errors
Projects have namespace: ... 
2/1 ... yes
4/2 ... yes
4/3 ... yes
4/4 ... yes
4/5 ... yes
3/6 ... yes
4/7 ... yes
4/8 ... yes
8/10 ... yes
3/11 ... yes
8/12 ... yes
3/13 ... yes
8/14 ... yes
5/15 ... yes
3/16 ... yes
8/17 ... yes
29/18 ... yes
4/19 ... yes
8/20 ... yes
5/21 ... yes
6/23 ... yes
Redis version >= 5.0.0? ... no
  Try fixing it:
  Your Redis version 4.0.14 is not supported anymore. Update your Redis server to a version >= 5.0.0
  For more information see:
  doc/administration/redis/index.html#redis-replication-and-failover-using-the-non-bundled-redis
  Please fix the error above and rerun the checks.
Ruby version >= 2.7.2 ? ... yes (2.7.4)
Git version >= 2.33.0 ? ... no
Your git bin path is "/usr/bin/git"
  Try fixing it:
  Update your git to a version >= 2.33.0 from Unknown
  Please fix the error above and rerun the checks.
Git user has default SSH configuration? ... yes
Active users: ... 16
Is authorized keys file accessible? ... skipped (authorized keys not enabled)
GitLab configured to store new projects in hashed storage? ... yes
All projects are in hashed storage? ... yes
Elasticsearch version 7.x (6.4 - 6.x deprecated to be removed in 13.8)? ... skipped (elasticsearch is disabled)

Checking GitLab App ... Finished


Checking GitLab subtasks ... Finished
Database Contents 
gitlabhq_production=> select * from terraform_state_versions;
-[ RECORD 1 ]------------+------------------------------
id                       | 15
terraform_state_id       | 4
created_by_user_id       | 2
created_at               | 2021-11-02 17:24:18.098171+00
updated_at               | 2021-11-02 17:24:18.098171+00
version                  | 0
file_store               | 1
file                     | 0.tfstate
verification_retry_count | 
verification_retry_at    | 
verified_at              | 
verification_checksum    | 
verification_failure     | 
ci_build_id              | 
verification_started_at  | 
verification_state       | 0

gitlabhq_production=> select * from terraform_states;
-[ RECORD 1 ]------+---------------------------------
id                 | 2
project_id         | 18
created_at         | 2021-11-01 20:57:26.820616+00
updated_at         | 2021-11-01 21:22:01.504613+00
file_store         |
file               |
lock_xid           |
locked_at          |
locked_by_user_id  |
uuid               | d3c2fd6c56ed8f18b5174414e20172de
name               | dev-sre
versioning_enabled | t
-[ RECORD 2 ]------+---------------------------------
id                 | 3
project_id         | 18
created_at         | 2021-11-01 21:10:15.196989+00
updated_at         | 2021-11-01 21:22:37.426851+00
file_store         |
file               |
lock_xid           |
locked_at          |
locked_by_user_id  |
uuid               | eaac7985f4fa48b0b74b53ed29a6f2a9
name               | dev-namespace
versioning_enabled | t
-[ RECORD 3 ]------+---------------------------------
id                 | 4
project_id         | 18
created_at         | 2021-11-02 17:24:14.324062+00
updated_at         | 2021-11-02 17:26:59.192122+00
file_store         |
file               |
lock_xid           |
locked_at          |
locked_by_user_id  |
uuid               | 971ef90b0d99743f70cf036de7da8c43
name               | dev-init
versioning_enabled | t
gitlabhq_production=> select * from terraform_state_versions_id_seq ;
-[ RECORD 1 ]--
last_value | 18
log_cnt    | 30
is_called  | t


gitlabhq_production=> select * from terraform_states_id_seq;
-[ RECORD 1 ]--
last_value | 4
log_cnt    | 32
is_called  | t

Possible fixes

Unsure.

Edited by Jan Martin Harris-Harasym