Gitlab Self Hosted gitlab-runner 15.5.0 (0d4137b8) Terraform Crash: attempt to write nil state to file

Summary

With no apparent changes to terraform code, gitlab server, gitlab runner, or gitlab image we are seeing a terraform crash immediately after successful initialization.

Terraform has been successfully initialized!
45!!!!!!!!!!!!!!!!!!!!!!!!!!! TERRAFORM CRASH !!!!!!!!!!!!!!!!!!!!!!!!!!!!
46Terraform crashed! This is always indicative of a bug within Terraform.
47Please report the crash with Terraform[1] so that we can fix this.
48When reporting bugs, please include your terraform version, the stack trace
49shown below, and any additional information which may help replicate the issue.
50[1]: https://github.com/hashicorp/terraform/issues
51!!!!!!!!!!!!!!!!!!!!!!!!!!! TERRAFORM CRASH !!!!!!!!!!!!!!!!!!!!!!!!!!!!
52attempt to write nil state to file
53goroutine 39 [running]:
54runtime/debug.Stack()
55	/usr/local/go/src/runtime/debug/stack.go:24 +0x65
56runtime/debug.PrintStack()
57	/usr/local/go/src/runtime/debug/stack.go:16 +0x19
58github.com/hashicorp/terraform/internal/logging.PanicHandler()
59	/home/circleci/project/project/internal/logging/panic.go:55 +0x153
60panic({0x2143f40, 0x2b43210})
61	/usr/local/go/src/runtime/panic.go:884 +0x212
62github.com/hashicorp/terraform/internal/states/statefile.writeStateV4(0xc00068ba8c?, {0x2b46420?, 0xc001052460?})
63	/home/circleci/project/project/internal/states/statefile/version4.go:322 +0x15f8
64github.com/hashicorp/terraform/internal/states/statefile.Write(0xc00068bc50?, {0x2b46420?, 0xc001052460?})
65	/home/circleci/project/project/internal/states/statefile/write.go:15 +0x45
66github.com/hashicorp/terraform/internal/plans/planfile.Create({0x7ffc16421625?, 0xa?}, {0xc000176320, 0xc00068be30, 0xc0015ac0c0, 0xc000fa61a0, 0xc000a272c0})
67	/home/circleci/project/project/internal/plans/planfile/writer.go:103 +0x670
68github.com/hashicorp/terraform/internal/backend/local.(*Local).opPlan(0xc000a1e960, {0x2b6adb8, 0xc000a4fb40}, {0x2b6adb8, 0xc000a4fb80}, 0xc000aaa480, 0xc000a4fb00)
69	/home/circleci/project/project/internal/backend/local/backend_plan.go:138 +0x905
70github.com/hashicorp/terraform/internal/backend/local.(*Local).Operation.func1()
71	/home/circleci/project/project/internal/backend/local/backend.go:323 +0xc3
72created by github.com/hashicorp/terraform/internal/backend/local.(*Local).Operation
73	/home/circleci/project/project/internal/backend/local/backend.go:316 +0x44a

Steps to reproduce

Unclear how to reproduce as we have not been able to identify any changes on our side. This is however blocking all terraform execution. We have tried to dump the Gitlab Managed Terraform State files and recreate, which happens successfully, but we still encounter the error.

Example Project

I will attempt to do this

What is the current bug behavior?

gitlab-ci.yml

image: registry.gitlab.com/gitlab-org/terraform-images/stable:latest

variables:
  TF_ROOT: ${CI_PROJECT_DIR}
  PACKAGE_TAG: "0.1"
  DEMO_RUN: "${CI_PIPELINE_ID}-${CI_COMMIT_SHORT_SHA}"
  ENV_PREFIX: "${GITLAB_USER_ID}-${DEMO_RUN}"
  BASE_PROJECT_DEPLOY: "eng-demo"
  TF_VAR_silvermint_version: "4403-5c3b1054"

before_script:
  - export GOOGLE_CREDENTIALS=$(echo $BASE64_GOOGLE_CREDENTIALS | base64 -d)
  - cd ${TF_ROOT}

stages:
  - init
  - personal
  - devnet
  - testnet
  - mainnet

init:
  stage: init
  rules:
    - when: always
  variables:
    TF_ADDRESS: "${CI_API_V4_URL}/projects/94/terraform/state/${BASE_PROJECT_DEPLOY}"
    TF_VAR_demo_run: "${DEMO_RUN}"
  script:
    - echo "pipeline source - $CI_PIPELINE_SOURCE"
    - echo "pipeline triggered - $CI_PIPELINE_TRIGGERED"
    - gitlab-terraform init
    - 'for f in *; do curl --header "JOB-TOKEN: ${CI_JOB_TOKEN}" --upload-file $f "${CI_API_V4_URL}/projects/94/packages/generic/demo/${PACKAGE_TAG}/$f"; done'

cleanup-personal:
  stage: personal
  needs: [init]
  rules:
    - if: '$CI_PROJECT_NAMESPACE != "eng"'
      when: on_success
  variables:
    TF_ADDRESS: "${CI_API_V4_URL}/projects/94/terraform/state/${BASE_PROJECT_DEPLOY}-${GITLAB_USER_ID}"
    TF_VAR_demo_run: "${DEMO_RUN}"
    TF_VAR_gitlab_environment: $GITLAB_USER_LOGIN
    TF_VAR_gitlab_user: $GITLAB_USER_LOGIN
  script:
    - gitlab-terraform plan -destroy -var-file="./vars/personal.tfvars"
    - gitlab-terraform apply -destroy

Output:

[0KRunning with gitlab-runner 15.5.0 (0d4137b8)[0;m
[0K  on gitlab-runner-igm-x2zw HUwKExYi[0;m
section_start:1668644149:resolve_secrets
[0K[0K[36;1mResolving secrets[0;m[0;m
section_end:1668644149:resolve_secrets
[0Ksection_start:1668644149:prepare_executor
[0K[0K[36;1mPreparing the "docker" executor[0;m[0;m
[0KUsing Docker executor with image registry.gitlab.com/gitlab-org/terraform-images/stable:latest ...[0;m
[0KPulling docker image registry.gitlab.com/gitlab-org/terraform-images/stable:latest ...[0;m
[0KUsing docker image sha256:b21b349d093533df371f9327020548c6655e11e984d431ff3e5a02079d0d9919 for registry.gitlab.com/gitlab-org/terraform-images/stable:latest with digest registry.gitlab.com/gitlab-org/terraform-images/stable@sha256:9b3ffea135beed7cf41b6fc0cff709c4844d431a79a56b08fc64e66ce8678ec1 ...[0;m
section_end:1668644151:prepare_executor
[0Ksection_start:1668644151:prepare_script
[0K[0K[36;1mPreparing environment[0;m[0;m
Running on runner-huwkexyi-project-94-concurrent-0 via gitlab-runner-igm-x2zw...
section_end:1668644151:prepare_script
[0Ksection_start:1668644151:get_sources
[0K[0K[36;1mGetting source from Git repository[0;m[0;m
[32;1mFetching changes with git depth set to 20...[0;m
Reinitialized existing Git repository in /builds/eng/demo/.git/
[32;1mChecking out 92306989 as dev...[0;m
Removing .terraform.lock.hcl
Removing .terraform/

[32;1mSkipping Git submodules setup[0;m
section_end:1668644152:get_sources
[0Ksection_start:1668644152:step_script
[0K[0K[36;1mExecuting "step_script" stage of the job script[0;m[0;m
[0KUsing docker image sha256:b21b349d093533df371f9327020548c6655e11e984d431ff3e5a02079d0d9919 for registry.gitlab.com/gitlab-org/terraform-images/stable:latest with digest registry.gitlab.com/gitlab-org/terraform-images/stable@sha256:9b3ffea135beed7cf41b6fc0cff709c4844d431a79a56b08fc64e66ce8678ec1 ...[0;m
[32;1m$ export GOOGLE_CREDENTIALS=$(echo $BASE64_GOOGLE_CREDENTIALS | base64 -d)[0;m
[32;1m$ cd ${TF_ROOT}[0;m
[32;1m$ gitlab-terraform plan -destroy -var-file="./vars/devnet.tfvars"[0;m

[0m[1mInitializing the backend...[0m
[0m[32m
Successfully configured the backend "http"! Terraform will automatically
use this backend unless the backend configuration changes.[0m

[0m[1mInitializing provider plugins...[0m
- Finding hashicorp/google versions matching "4.43.0"...
- Finding latest version of hashicorp/kubernetes...
- Finding latest version of hashicorp/helm...
- Installing hashicorp/google v4.43.0...
- Installed hashicorp/google v4.43.0 (signed by HashiCorp)
- Installing hashicorp/kubernetes v2.15.0...
- Installed hashicorp/kubernetes v2.15.0 (signed by HashiCorp)
- Installing hashicorp/helm v2.7.1...
- Installed hashicorp/helm v2.7.1 (signed by HashiCorp)

Terraform has created a lock file [1m.terraform.lock.hcl[0m to record the provider
selections it made above. Include this file in your version control repository
so that Terraform can guarantee to make the same selections by default when
you run "terraform init" in the future.[0m

[0m[1m[32mTerraform has been successfully initialized![0m[32m[0m

!!!!!!!!!!!!!!!!!!!!!!!!!!! TERRAFORM CRASH !!!!!!!!!!!!!!!!!!!!!!!!!!!!

Terraform crashed! This is always indicative of a bug within Terraform.
Please report the crash with Terraform[1] so that we can fix this.

When reporting bugs, please include your terraform version, the stack trace
shown below, and any additional information which may help replicate the issue.

[1]: https://github.com/hashicorp/terraform/issues

!!!!!!!!!!!!!!!!!!!!!!!!!!! TERRAFORM CRASH !!!!!!!!!!!!!!!!!!!!!!!!!!!!

attempt to write nil state to file
goroutine 11 [running]:
runtime/debug.Stack()
	/usr/local/go/src/runtime/debug/stack.go:24 +0x65
runtime/debug.PrintStack()
	/usr/local/go/src/runtime/debug/stack.go:16 +0x19
github.com/hashicorp/terraform/internal/logging.PanicHandler()
	/home/circleci/project/project/internal/logging/panic.go:55 +0x153
panic({0x2143f40, 0x2b43210})
	/usr/local/go/src/runtime/panic.go:884 +0x212
github.com/hashicorp/terraform/internal/states/statefile.writeStateV4(0xc000a8fa8c?, {0x2b46420?, 0xc000f41540?})
	/home/circleci/project/project/internal/states/statefile/version4.go:322 +0x15f8
github.com/hashicorp/terraform/internal/states/statefile.Write(0xc000a8fc50?, {0x2b46420?, 0xc000f41540?})
	/home/circleci/project/project/internal/states/statefile/write.go:15 +0x45
github.com/hashicorp/terraform/internal/plans/planfile.Create({0x7ffe3a90d629?, 0xa?}, {0xc000142e78, 0xc000a8fe30, 0xc001cdc330, 0xc00216d110, 0xc000991ec0})
	/home/circleci/project/project/internal/plans/planfile/writer.go:103 +0x670
github.com/hashicorp/terraform/internal/backend/local.(*Local).opPlan(0xc0009c2280, {0x2b6adb8, 0xc0009b92c0}, {0x2b6adb8, 0xc0009b9300}, 0xc00098aa20, 0xc0009b9280)
	/home/circleci/project/project/internal/backend/local/backend_plan.go:138 +0x905
github.com/hashicorp/terraform/internal/backend/local.(*Local).Operation.func1()
	/home/circleci/project/project/internal/backend/local/backend.go:323 +0xc3
created by github.com/hashicorp/terraform/internal/backend/local.(*Local).Operation
	/home/circleci/project/project/internal/backend/local/backend.go:316 +0x44a
section_end:1668644160:step_script
[0Ksection_start:1668644160:cleanup_file_variables
[0K[0K[36;1mCleaning up project directory and file based variables[0;m[0;m
section_end:1668644161:cleanup_file_variables
[0K[31;1mERROR: Job failed: exit code 11
[0;m

What is the expected correct behavior?

Output from Same Stage on 2022-11-15 4:44pm PST:

[0KRunning with gitlab-runner 15.5.0 (0d4137b8)[0;m
[0K  on gitlab-runner-igm-x2zw HUwKExYi[0;m
section_start:1668559388:resolve_secrets
[0K[0K[36;1mResolving secrets[0;m[0;m
section_end:1668559388:resolve_secrets
[0Ksection_start:1668559388:prepare_executor
[0K[0K[36;1mPreparing the "docker" executor[0;m[0;m
[0KUsing Docker executor with image registry.gitlab.com/gitlab-org/terraform-images/stable:latest ...[0;m
[0KPulling docker image registry.gitlab.com/gitlab-org/terraform-images/stable:latest ...[0;m
[0KUsing docker image sha256:b21b349d093533df371f9327020548c6655e11e984d431ff3e5a02079d0d9919 for registry.gitlab.com/gitlab-org/terraform-images/stable:latest with digest registry.gitlab.com/gitlab-org/terraform-images/stable@sha256:9b3ffea135beed7cf41b6fc0cff709c4844d431a79a56b08fc64e66ce8678ec1 ...[0;m
section_end:1668559390:prepare_executor
[0Ksection_start:1668559390:prepare_script
[0K[0K[36;1mPreparing environment[0;m[0;m
Running on runner-huwkexyi-project-94-concurrent-0 via gitlab-runner-igm-x2zw...
section_end:1668559390:prepare_script
[0Ksection_start:1668559390:get_sources
[0K[0K[36;1mGetting source from Git repository[0;m[0;m
[32;1mFetching changes with git depth set to 20...[0;m
Reinitialized existing Git repository in /builds/eng/demo/.git/
[32;1mChecking out 92306989 as dev...[0;m
Removing .terraform.lock.hcl
Removing .terraform/
Removing plan.cache
Removing plan.json

[32;1mSkipping Git submodules setup[0;m
section_end:1668559391:get_sources
[0Ksection_start:1668559391:step_script
[0K[0K[36;1mExecuting "step_script" stage of the job script[0;m[0;m
[0KUsing docker image sha256:b21b349d093533df371f9327020548c6655e11e984d431ff3e5a02079d0d9919 for registry.gitlab.com/gitlab-org/terraform-images/stable:latest with digest registry.gitlab.com/gitlab-org/terraform-images/stable@sha256:9b3ffea135beed7cf41b6fc0cff709c4844d431a79a56b08fc64e66ce8678ec1 ...[0;m
[32;1m$ export GOOGLE_CREDENTIALS=$(echo $BASE64_GOOGLE_CREDENTIALS | base64 -d)[0;m
[32;1m$ cd ${TF_ROOT}[0;m
[32;1m$ gitlab-terraform plan -destroy -var-file="./vars/devnet.tfvars"[0;m

[0m[1mInitializing the backend...[0m
[0m[32m
Successfully configured the backend "http"! Terraform will automatically
use this backend unless the backend configuration changes.[0m

[0m[1mInitializing provider plugins...[0m
- Finding hashicorp/google versions matching "4.43.0"...
- Finding latest version of hashicorp/helm...
- Finding latest version of hashicorp/kubernetes...
- Installing hashicorp/google v4.43.0...
- Installed hashicorp/google v4.43.0 (signed by HashiCorp)
- Installing hashicorp/helm v2.7.1...
- Installed hashicorp/helm v2.7.1 (signed by HashiCorp)
- Installing hashicorp/kubernetes v2.15.0...
- Installed hashicorp/kubernetes v2.15.0 (signed by HashiCorp)

Terraform has created a lock file [1m.terraform.lock.hcl[0m to record the provider
selections it made above. Include this file in your version control repository
so that Terraform can guarantee to make the same selections by default when
you run "terraform init" in the future.[0m

[0m[1m[32mTerraform has been successfully initialized![0m[32m[0m
[0m[1mdata.google_compute_address.graylog-ext-ip: Reading...[0m[0m
[0m[1mdata.google_redis_instance.redis: Reading...[0m[0m
[0m[1mdata.google_compute_image.node-compute-image: Reading...[0m[0m
[0m[1mdata.google_compute_address.prometheus-ext-ip: Reading...[0m[0m
[0m[1mgoogle_compute_target_pool.sender-target-pool: Refreshing state... [id=projects/corp-331300/regions/us-west3/targetPools/devnet-demo-sender-target-pool][0m
[0m[1mdata.google_compute_address.graylog-int-ip: Reading...[0m[0m
[0m[1mgoogle_service_account.sender_service_account: Refreshing state... [id=projects/corp-331300/serviceAccounts/devnet-sender@corp-331300.iam.gserviceaccount.com][0m
[0m[1mgoogle_compute_target_pool.node-target-pool: Refreshing state... [id=projects/corp-331300/regions/us-west3/targetPools/devnet-demo-node-target-pool][0m
[0m[1mdata.google_compute_image.sender-compute-image: Reading...[0m[0m
[0m[1mdata.google_compute_address.grafana-ext-ip: Reading...[0m[0m
[0m[1mdata.google_client_config.current: Reading...[0m[0m
[0m[1mdata.google_client_config.current: Read complete after 0s [id=projects/corp-331300/regions//zones/][0m
[0m[1mdata.google_compute_network.network: Reading...[0m[0m
[0m[1mdata.google_compute_address.grafana-ext-ip: Read complete after 1s [id=projects/corp-331300/regions/us-west3/addresses/corp-grafana-ext-ip][0m
[0m[1mdata.google_compute_subnetwork.subnetwork: Reading...[0m[0m
[0m[1mdata.google_compute_address.prometheus-ext-ip: Read complete after 1s [id=projects/corp-331300/regions/us-west3/addresses/corp-prometheus-ext-ip][0m
[0m[1mgoogle_service_account.node_service_account: Refreshing state... [id=projects/corp-331300/serviceAccounts/devnet-node@corp-331300.iam.gserviceaccount.com][0m
[0m[1mdata.google_compute_address.graylog-ext-ip: Read complete after 1s [id=projects/corp-331300/regions/us-west3/addresses/corp-graylog-ext-ip][0m
[0m[1mgoogle_project_iam_member.sender_storage_admin: Refreshing state... [id=corp-331300/roles/storage.admin/serviceAccount:devnet-sender@corp-331300.iam.gserviceaccount.com][0m
[0m[1mdata.google_compute_address.graylog-int-ip: Read complete after 1s [id=projects/corp-331300/regions/us-west3/addresses/corp-graylog-int-ip][0m
[0m[1mdata.google_redis_instance.redis: Read complete after 1s [id=projects/corp-331300/locations/us-west3/instances/corp-redis][0m
[0m[1mgoogle_project_iam_member.node_storage_admin: Refreshing state... [id=corp-331300/roles/storage.admin/serviceAccount:devnet-node@corp-331300.iam.gserviceaccount.com][0m
[0m[1mdata.google_compute_image.node-compute-image: Read complete after 1s [id=projects/debian-cloud/global/images/debian-11-bullseye-v20221102][0m
[0m[1mdata.google_compute_image.sender-compute-image: Read complete after 1s [id=projects/debian-cloud/global/images/debian-11-bullseye-v20221102][0m
[0m[1mdata.google_compute_subnetwork.subnetwork: Read complete after 0s [id=projects/corp-331300/regions/us-west3/subnetworks/corp-subnetwork][0m
[0m[1mdata.google_compute_network.network: Read complete after 0s [id=projects/corp-331300/global/networks/corp-network][0m
[0m[1mgoogle_compute_instance_template.sender-instance-template: Refreshing state... [id=projects/corp-331300/global/instanceTemplates/devnet-demo-sender-5156-d29fe174-20221116002753522600000001][0m
[0m[1mgoogle_compute_instance_template.node-instance-template: Refreshing state... [id=projects/corp-331300/global/instanceTemplates/devnet-demo-node-5156-d29fe174-20221116002753539100000002][0m
[0m[1mdata.google_compute_router.router: Reading...[0m[0m
[0m[1mdata.google_compute_router.router: Read complete after 0s [id=corp-router][0m
[0m[1mgoogle_compute_instance_group_manager.sender-instance-group-manager: Refreshing state... [id=projects/corp-331300/zones/us-west3-a/instanceGroupManagers/devnet-demo-sender-igm][0m
[0m[1mgoogle_compute_instance_group_manager.node-instance-group-manager: Refreshing state... [id=projects/corp-331300/zones/us-west3-a/instanceGroupManagers/devnet-demo-node-igm][0m
[0m[1mgoogle_compute_autoscaler.sender-autoscaler: Refreshing state... [id=projects/corp-331300/zones/us-west3-a/autoscalers/devnet-demo-sender-autoscaler][0m
[0m[1mgoogle_compute_autoscaler.node-autoscaler: Refreshing state... [id=projects/corp-331300/zones/us-west3-a/autoscalers/devnet-demo-node-autoscaler][0m

Terraform used the selected providers to generate the following execution
...

Relevant logs and/or screenshots

[0KRunning with gitlab-runner 15.5.0 (0d4137b8)[0;m
[0K  on gitlab-runner-igm-x2zw HUwKExYi[0;m
section_start:1668644149:resolve_secrets
[0K[0K[36;1mResolving secrets[0;m[0;m
section_end:1668644149:resolve_secrets
[0Ksection_start:1668644149:prepare_executor
[0K[0K[36;1mPreparing the "docker" executor[0;m[0;m
[0KUsing Docker executor with image registry.gitlab.com/gitlab-org/terraform-images/stable:latest ...[0;m
[0KPulling docker image registry.gitlab.com/gitlab-org/terraform-images/stable:latest ...[0;m
[0KUsing docker image sha256:b21b349d093533df371f9327020548c6655e11e984d431ff3e5a02079d0d9919 for registry.gitlab.com/gitlab-org/terraform-images/stable:latest with digest registry.gitlab.com/gitlab-org/terraform-images/stable@sha256:9b3ffea135beed7cf41b6fc0cff709c4844d431a79a56b08fc64e66ce8678ec1 ...[0;m
section_end:1668644151:prepare_executor
[0Ksection_start:1668644151:prepare_script
[0K[0K[36;1mPreparing environment[0;m[0;m
Running on runner-huwkexyi-project-94-concurrent-0 via gitlab-runner-igm-x2zw...
section_end:1668644151:prepare_script
[0Ksection_start:1668644151:get_sources
[0K[0K[36;1mGetting source from Git repository[0;m[0;m
[32;1mFetching changes with git depth set to 20...[0;m
Reinitialized existing Git repository in /builds/eng/demo/.git/
[32;1mChecking out 92306989 as dev...[0;m
Removing .terraform.lock.hcl
Removing .terraform/

[32;1mSkipping Git submodules setup[0;m
section_end:1668644152:get_sources
[0Ksection_start:1668644152:step_script
[0K[0K[36;1mExecuting "step_script" stage of the job script[0;m[0;m
[0KUsing docker image sha256:b21b349d093533df371f9327020548c6655e11e984d431ff3e5a02079d0d9919 for registry.gitlab.com/gitlab-org/terraform-images/stable:latest with digest registry.gitlab.com/gitlab-org/terraform-images/stable@sha256:9b3ffea135beed7cf41b6fc0cff709c4844d431a79a56b08fc64e66ce8678ec1 ...[0;m
[32;1m$ export GOOGLE_CREDENTIALS=$(echo $BASE64_GOOGLE_CREDENTIALS | base64 -d)[0;m
[32;1m$ cd ${TF_ROOT}[0;m
[32;1m$ gitlab-terraform plan -destroy -var-file="./vars/devnet.tfvars"[0;m

[0m[1mInitializing the backend...[0m
[0m[32m
Successfully configured the backend "http"! Terraform will automatically
use this backend unless the backend configuration changes.[0m

[0m[1mInitializing provider plugins...[0m
- Finding hashicorp/google versions matching "4.43.0"...
- Finding latest version of hashicorp/kubernetes...
- Finding latest version of hashicorp/helm...
- Installing hashicorp/google v4.43.0...
- Installed hashicorp/google v4.43.0 (signed by HashiCorp)
- Installing hashicorp/kubernetes v2.15.0...
- Installed hashicorp/kubernetes v2.15.0 (signed by HashiCorp)
- Installing hashicorp/helm v2.7.1...
- Installed hashicorp/helm v2.7.1 (signed by HashiCorp)

Terraform has created a lock file [1m.terraform.lock.hcl[0m to record the provider
selections it made above. Include this file in your version control repository
so that Terraform can guarantee to make the same selections by default when
you run "terraform init" in the future.[0m

[0m[1m[32mTerraform has been successfully initialized![0m[32m[0m

!!!!!!!!!!!!!!!!!!!!!!!!!!! TERRAFORM CRASH !!!!!!!!!!!!!!!!!!!!!!!!!!!!

Terraform crashed! This is always indicative of a bug within Terraform.
Please report the crash with Terraform[1] so that we can fix this.

When reporting bugs, please include your terraform version, the stack trace
shown below, and any additional information which may help replicate the issue.

[1]: https://github.com/hashicorp/terraform/issues

!!!!!!!!!!!!!!!!!!!!!!!!!!! TERRAFORM CRASH !!!!!!!!!!!!!!!!!!!!!!!!!!!!

attempt to write nil state to file
goroutine 11 [running]:
runtime/debug.Stack()
	/usr/local/go/src/runtime/debug/stack.go:24 +0x65
runtime/debug.PrintStack()
	/usr/local/go/src/runtime/debug/stack.go:16 +0x19
github.com/hashicorp/terraform/internal/logging.PanicHandler()
	/home/circleci/project/project/internal/logging/panic.go:55 +0x153
panic({0x2143f40, 0x2b43210})
	/usr/local/go/src/runtime/panic.go:884 +0x212
github.com/hashicorp/terraform/internal/states/statefile.writeStateV4(0xc000a8fa8c?, {0x2b46420?, 0xc000f41540?})
	/home/circleci/project/project/internal/states/statefile/version4.go:322 +0x15f8
github.com/hashicorp/terraform/internal/states/statefile.Write(0xc000a8fc50?, {0x2b46420?, 0xc000f41540?})
	/home/circleci/project/project/internal/states/statefile/write.go:15 +0x45
github.com/hashicorp/terraform/internal/plans/planfile.Create({0x7ffe3a90d629?, 0xa?}, {0xc000142e78, 0xc000a8fe30, 0xc001cdc330, 0xc00216d110, 0xc000991ec0})
	/home/circleci/project/project/internal/plans/planfile/writer.go:103 +0x670
github.com/hashicorp/terraform/internal/backend/local.(*Local).opPlan(0xc0009c2280, {0x2b6adb8, 0xc0009b92c0}, {0x2b6adb8, 0xc0009b9300}, 0xc00098aa20, 0xc0009b9280)
	/home/circleci/project/project/internal/backend/local/backend_plan.go:138 +0x905
github.com/hashicorp/terraform/internal/backend/local.(*Local).Operation.func1()
	/home/circleci/project/project/internal/backend/local/backend.go:323 +0xc3
created by github.com/hashicorp/terraform/internal/backend/local.(*Local).Operation
	/home/circleci/project/project/internal/backend/local/backend.go:316 +0x44a
section_end:1668644160:step_script
[0Ksection_start:1668644160:cleanup_file_variables
[0K[0K[36;1mCleaning up project directory and file based variables[0;m[0;m
section_end:1668644161:cleanup_file_variables
[0K[31;1mERROR: Job failed: exit code 11
[0;m

Output of checks

Results of GitLab environment info

Expand for output related to GitLab environment info

root@git:~# sudo gitlab-rake gitlab:env:info

System information
System:         Debian 10
Proxy:          no
Current User:   git
Using RVM:      no
Ruby Version:   2.7.5p203
Gem Version:    3.1.6
Bundler Version:2.3.15
Rake Version:   13.0.6
Redis Version:  6.2.7
Sidekiq Version:6.4.2
Go Version:     unknown

GitLab information
Version:        15.5.4-ee
Revision:       d3dda7548e0
Directory:      /opt/gitlab/embedded/service/gitlab-rails
DB Adapter:     PostgreSQL
DB Version:     12.10
URL:            https://git.pyro.cloud
HTTP Clone URL: https://git.pyro.cloud/some-group/some-project.git
SSH Clone URL:  git@git.pyro.cloud:some-group/some-project.git
Elasticsearch:  no
Geo:            no
Using LDAP:     no
Using Omniauth: yes
Omniauth Providers: google_oauth2

GitLab Shell
Version:        14.12.0
Repository storage paths:
- default:      /var/opt/gitlab/git-data/repositories
GitLab Shell path:              /opt/gitlab/embedded/service/gitlab-shell

Results of GitLab application Check

Expand for output related to the GitLab application check

root@git:~# sudo gitlab-rake gitlab:check SANITIZE=true
Checking GitLab subtasks ...
Checking GitLab Shell ...
GitLab Shell: ... GitLab Shell version >= 14.12.0 ? ... OK (14.12.0)
Running /opt/gitlab/embedded/service/gitlab-shell/bin/check
Internal API available: OK
Redis available via internal API: OK
gitlab-shell self-check successful
Checking GitLab Shell ... Finished
Checking Gitaly ...
Gitaly: ... default ... OK
Checking Gitaly ... Finished
Checking Sidekiq ...
Sidekiq: ... Running? ... yes
Number of Sidekiq processes (cluster/worker) ... 1/1
Checking Sidekiq ... Finished
Checking Incoming Email ...
Incoming Email: ... Reply by email is disabled in config/gitlab.yml
Checking Incoming Email ... Finished
Checking LDAP ...
LDAP: ... LDAP is disabled in config/gitlab.yml
Checking LDAP ... Finished
Checking GitLab App ...
Database config exists? ... yes
All migrations up? ... yes
Database contains orphaned GroupMembers? ... no
GitLab config exists? ... yes
GitLab config up to date? ... yes
Log directory writable? ... yes
Tmp directory writable? ... yes
Uploads directory exists? ... yes
Uploads directory has correct permissions? ... yes
Uploads directory tmp has correct permissions? ... yes
Systemd unit files or init script exist? ... skipped (omnibus-gitlab has neither init script nor systemd units)
Systemd unit files or init script up-to-date? ... skipped (omnibus-gitlab has neither init script nor systemd units)
Projects have namespace: ...
2/1 ... yes
6/7 ... yes
6/8 ... yes
8/9 ... yes
7/10 ... yes
7/11 ... yes
7/12 ... yes
4/13 ... yes
4/14 ... yes
4/15 ... yes
4/16 ... yes
4/17 ... yes
4/18 ... yes
4/19 ... yes
4/20 ... yes
4/21 ... yes
4/22 ... yes
4/23 ... yes
4/24 ... yes
9/25 ... yes
9/26 ... yes
3/27 ... yes
3/28 ... yes
3/29 ... yes
3/30 ... yes
3/31 ... yes
3/32 ... yes
3/33 ... yes
3/34 ... yes
3/35 ... yes
135/36 ... yes
5/37 ... yes
5/38 ... yes
11/39 ... yes
12/40 ... yes
12/41 ... yes
13/42 ... yes
14/43 ... yes
14/44 ... yes
14/45 ... yes
15/46 ... yes
16/47 ... yes
22/48 ... yes
22/49 ... yes
22/50 ... yes
22/51 ... yes
22/52 ... yes
24/55 ... yes
3/56 ... yes
4/57 ... yes
4/58 ... yes
24/60 ... yes
4/61 ... yes
22/62 ... yes
27/67 ... yes
28/68 ... yes
28/69 ... yes
4/70 ... yes
22/71 ... yes
29/72 ... yes
24/74 ... yes
28/75 ... yes
3/76 ... yes
29/77 ... yes
29/78 ... yes
29/80 ... yes
4/82 ... yes
3/83 ... yes
3/84 ... yes
4/85 ... yes
135/86 ... yes
4/87 ... yes
4/89 ... yes
124/90 ... yes
130/92 ... yes
130/93 ... yes
4/94 ... yes
124/97 ... yes
24/99 ... yes
27/101 ... yes
4/102 ... yes
124/103 ... yes
24/104 ... yes
4/105 ... yes
4/107 ... yes
124/108 ... yes
4/109 ... yes
27/110 ... yes
116/114 ... yes
116/116 ... yes
115/117 ... yes
115/118 ... yes
115/119 ... yes
116/120 ... yes
4/123 ... yes
116/124 ... yes
3/126 ... yes
3/127 ... yes
124/130 ... yes
124/132 ... yes
4/133 ... yes
4/134 ... yes
22/135 ... yes
29/136 ... yes
4/137 ... yes
116/138 ... yes
4/139 ... yes
4/140 ... yes
22/141 ... yes
4/143 ... yes
22/144 ... yes
124/145 ... yes
4/146 ... yes
22/148 ... yes
24/149 ... yes
116/150 ... yes
29/151 ... yes
3/152 ... yes
4/153 ... yes
24/155 ... yes
29/156 ... yes
116/157 ... yes
116/158 ... yes
116/159 ... yes
4/160 ... yes
116/161 ... yes
4/162 ... yes
24/163 ... yes
212/164 ... yes
4/165 ... yes
24/166 ... yes
3/167 ... yes
212/168 ... yes
212/169 ... yes
22/170 ... yes
4/172 ... yes
4/173 ... yes
Redis version >= 6.0.0? ... yes
Ruby version >= 2.7.2 ? ... yes (2.7.5)
Git user has default SSH configuration? ... yes
Active users: ... 23
Is authorized keys file accessible? ... yes
GitLab configured to store new projects in hashed storage? ... yes
All projects are in hashed storage? ... yes
Elasticsearch version 7.x-8.x or OpenSearch version 1.x ... skipped (Advanced Search is disabled)
Checking GitLab App ... Finished
Checking GitLab subtasks ... Finished

Possible fixes