Terraform state lock in CI "endpoint requires auth" for plan
Summary
I'm not able to lock terraform state files in CI for a terraform apply
of a previously create plan. The gitlab managed state store returns HTTP remote state endpoint requires auth
plan
and apply
in two steps (removing .terraform
dir) locally using my access token and gitlab username.[nic@ARCHY] ~/git/njdart-ccl/terraform-state-test/infrastructure rm -rf nic-test.plan .terraform
[nic@ARCHY] ~/git/njdart-ccl/terraform-state-test/infrastructure eval "terraform init `cat tfinit.conf`"
Initializing the backend...
Successfully configured the backend "http"! Terraform will automatically
use this backend unless the backend configuration changes.
2021/08/13 16:17:46 [DEBUG] GET https://gitlab.com/api/v4/projects/28843863/terraform/state/nic-test
Initializing provider plugins...
- Reusing previous version of hashicorp/random from the dependency lock file
- Installing hashicorp/random v3.1.0...
- Installed hashicorp/random v3.1.0 (signed by HashiCorp)
Terraform has been successfully initialized!
You may now begin working with Terraform. Try running "terraform plan" to see
any changes that are required for your infrastructure. All Terraform commands
should now work.
If you ever set or change modules or backend configuration for Terraform,
rerun this command to reinitialize your working directory. If you forget, other
commands will detect it and remind you to do so if necessary.
[nic@ARCHY] ~/git/njdart-ccl/terraform-state-test/infrastructure terraform plan -out nic-test.plan
2021/08/13 16:18:18 [DEBUG] POST https://gitlab.com/api/v4/projects/28843863/terraform/state/nic-test/lock
Acquiring state lock. This may take a few moments...
2021/08/13 16:18:19 [DEBUG] GET https://gitlab.com/api/v4/projects/28843863/terraform/state/nic-test
random_pet.example: Refreshing state... [id=highly-welcome-escargot]
Terraform used the selected providers to generate the following execution plan. Resource actions are indicated with the following symbols:
-/+ destroy and then create replacement
Terraform will perform the following actions:
# random_pet.example must be replaced
-/+ resource "random_pet" "example" {
~ id = "highly-welcome-escargot" -> (known after apply)
~ length = 3 -> 5 # forces replacement
# (1 unchanged attribute hidden)
}
Plan: 1 to add, 0 to change, 1 to destroy.
Changes to Outputs:
~ pet-names = "highly-welcome-escargot" -> (known after apply)
────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────
Saved the plan to: nic-test.plan
To perform exactly these actions, run the following command to apply:
terraform apply "nic-test.plan"
2021/08/13 16:18:20 [DEBUG] DELETE https://gitlab.com/api/v4/projects/28843863/terraform/state/nic-test/lock
[nic@ARCHY] ~/git/njdart-ccl/terraform-state-test/infrastructure rm -rf .terraform
[nic@ARCHY] ~/git/njdart-ccl/terraform-state-test/infrastructure eval "terraform init `cat tfinit.conf`"
Initializing the backend...
Successfully configured the backend "http"! Terraform will automatically
use this backend unless the backend configuration changes.
2021/08/13 16:18:28 [DEBUG] GET https://gitlab.com/api/v4/projects/28843863/terraform/state/nic-test
Initializing provider plugins...
- Reusing previous version of hashicorp/random from the dependency lock file
- Installing hashicorp/random v3.1.0...
- Installed hashicorp/random v3.1.0 (signed by HashiCorp)
Terraform has been successfully initialized!
You may now begin working with Terraform. Try running "terraform plan" to see
any changes that are required for your infrastructure. All Terraform commands
should now work.
If you ever set or change modules or backend configuration for Terraform,
rerun this command to reinitialize your working directory. If you forget, other
commands will detect it and remind you to do so if necessary.
[nic@ARCHY] ~/git/njdart-ccl/terraform-state-test/infrastructure terraform apply nic-test.plan
2021/08/13 16:18:35 [DEBUG] POST https://gitlab.com/api/v4/projects/28843863/terraform/state/nic-test/lock
2021/08/13 16:18:35 [DEBUG] GET https://gitlab.com/api/v4/projects/28843863/terraform/state/nic-test
random_pet.example: Destroying... [id=highly-welcome-escargot]
random_pet.example: Destruction complete after 0s
random_pet.example: Creating...
random_pet.example: Creation complete after 0s [id=needlessly-highly-allegedly-great-sponge]
2021/08/13 16:18:36 [DEBUG] POST https://gitlab.com/api/v4/projects/28843863/terraform/state/nic-test?ID=a7daa27d-053c-2236-3607-ba901d1bda42
2021/08/13 16:18:37 [DEBUG] DELETE https://gitlab.com/api/v4/projects/28843863/terraform/state/nic-test/lock
Apply complete! Resources: 1 added, 0 changed, 1 destroyed.
Outputs:
pet-names = "needlessly-highly-allegedly-great-sponge"
I am able to perform multiple plans in consecutive stages. (This was to test if the $CI_JOB_TOKEN
was becoming invalid is subsequent stages.
I am able to apply changes if I condense the plan and apply steps into a single operation
Steps to reproduce
In CI:
- First job:
terraform init -reconfigure -backend-config="address=...
-
terraform plan -lock=true -out=my-deployment.plan
aquires lock, produces plan, succeeds
- Second job
- Depends on the plan output
my-deployment.plan
from the first job -
terraform init -reconfigure -backend-config="address=...
same as first job) -
terraform apply my-deployment.plan
which fails trying to aquire the lock HTTP remote state endpoint requires auth.
- Depends on the plan output
See the job logs for first job and job logs for second job
I've checked the following:
- that i'm a maintainer of the project as per the docs
- that #219460 (closed) does not occur here
- that https://github.com/gitlabhq/terraform-provider-gitlab/issues/476 does not occur here
Example Project
This has occurred in other projects, but this repo reproduces the bug: https://gitlab.com/njdart-ccl/terraform-state-test
- The
main
branch has the bug present - The
double-plan
branch does not exhibit the bug when two consecutive stages perform a plan - The
condense-plan-apply
branch does not exhibit the bug if the plan and apply are not separated into two jobs
What is the current bug behavior?
State cannot be locked for terraform apply
when using a plan created by a previous job
What is the expected correct behavior?
State can be locked to apply a plan created by a previous job
Relevant logs and/or screenshots
https://gitlab.com/njdart-ccl/terraform-state-test/-/jobs/1502447711
Output of checks
This bug happens on GitLab.com (14.2.0-pre 1dcd275fbad
)
I have access to a private hosted gitlab instance (13.7.4-ee
, gitlab-runner 12.10.0~cc.135.g425cd11e (425cd11e)
) that also exhibits this behaviour
Possible fixes
As a workaround the plan and apply can be merged into a single step. This is sub-optimal as we wish to manually apply for some branches on projects I work on elsewhere