[Investigate]GitLab Managed Terraform State fails auth with CI_JOB_TOKEN
Summary
After upgrading to GitLab version 14.1.2, using ${CI_JOB_TOKEN}
fails authorization in a CI pipeline job. This was submitted on behalf of a US Federal customer (internal link only), so information has been redacted from the logs.
Steps to reproduce
Create a job that uses ${CI_JOB_TOKEN}
to authorized with the Terraform state backend
Example Project
An example project wasn't created, however, here is the .gitlab-ci.yml
file contents:
workflow:
rules:
- if: $CI_COMMIT_BRANCH
image:
name: hashicorp/terraform:light
entrypoint:
- '/usr/bin/env'
- 'PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin'
# Default output file for Terraform plan
variables:
PLAN: plan.tfplan
cache:
paths:
- .terraform
before_script:
# Set up state file per environment
- export GITLAB_TF_ADDRESS="${CI_API_V4_URL}/projects/${CI_PROJECT_ID}/terraform/state/${CI_ENVIRONMENT_SLUG}"
- echo $GITLAB_TF_ADDRESS
# Append bucket name with the environment name
- export TF_VAR_BUCKET_NAME="$TF_VAR_BUCKET_NAME-${CI_ENVIRONMENT_SLUG}"
# Write terraform http backend definition to a file as it cannot be a var
- echo -e "terraform { \n backend \"http\" {}\n}" >> http-backend.tf
- terraform --version
- |
terraform init -reconfigure \
-backend-config="username=gitlab-ci-token" \
-backend-config="password=${CI_JOB_TOKEN}" \
-backend-config="address=${GITLAB_TF_ADDRESS}" \
-backend-config="lock_address=${GITLAB_TF_ADDRESS}/lock" \
-backend-config="unlock_address=${GITLAB_TF_ADDRESS}/unlock"
stages:
- validate
- plan
- apply
- destroy
validate:
stage: validate
script: terraform validate
.plan_tpl:
stage: plan
dependencies: [validate]
script: terraform plan -lock=false -out=$PLAN
rules:
- if: $CI_COMMIT_BRANCH == 'development'
when: on_success
- if: $CI_COMMIT_BRANCH == 'staging'
when: on_success
- if: $CI_COMMIT_BRANCH == 'production'
when: on_success
- when: never
artifacts:
name: plan
paths:
- $PLAN
.apply_tpl:
stage: apply
script: terraform apply -lock=false -input=false $PLAN
rules:
- if: $CI_COMMIT_BRANCH == 'development'
when: manual
- if: $CI_COMMIT_BRANCH == 'staging'
when: manual
- if: $CI_COMMIT_BRANCH == 'production'
when: manual
- when: never
.destroy_tpl:
stage: destroy
script: terraform destroy -lock=false -auto-approve
rules:
- if: $CI_COMMIT_BRANCH == 'development' && $DESTROY_JOBS_ENABLED
when: manual
- if: $CI_COMMIT_BRANCH == 'staging' && $DESTROY_JOBS_ENABLED
when: manual
- if: $CI_COMMIT_BRANCH == 'production' && $DESTROY_JOBS_ENABLED
when: manual
- when: never
plan_primary:
extends: [ .plan_tpl]
environment:
name: primary-${CI_COMMIT_BRANCH}
action: prepare
apply_primary:
extends: [ .apply_tpl ]
environment:
name: primary-${CI_COMMIT_BRANCH}
on_stop: destroy_primary
needs: [plan_primary]
destroy_primary:
extends: [ .destroy_tpl ]
environment:
name: primary-${CI_COMMIT_BRANCH}
action: stop
needs: [apply_primary, plan_primary]
plan_secondary:
extends: [ .plan_tpl]
environment:
name: secondary-${CI_COMMIT_BRANCH}
action: prepare
apply_secondary:
extends: [ .apply_tpl ]
environment:
name: secondary-${CI_COMMIT_BRANCH}
on_stop: destroy_secondary
needs: [plan_secondary]
destroy_secondary:
extends: [ .destroy_tpl ]
environment:
name: secondary-${CI_COMMIT_BRANCH}
action: stop
needs: [apply_secondary, plan_secondary]
What is the current bug behavior?
Using ${CI_JOB_TOKEN}
to authenticate to the terraform managed state backend fails, however, using a personal access token and project access token authenticated to the terraform managed state backend successfully.
What is the expected correct behavior?
The CI job should be able to authorized with the Terraform state backend
Relevant logs and/or screenshots
Running with gitlab-runner 14.1.0 (8925d9a0)
on hir-earch-group Wzx5nsQV
section_start:1632764922:resolve_secrets^MResolving secrets
section_end:1632764922:resolve_secrets^Msection_start:1632764922:prepare_executor^MPreparing the "docker" executor
Using Docker executor with image [REDACTED]/base-images/utils/terraform:latest ...
Pulling docker image [REDACTED]/base-images/utils/terraform:latest ...
Using docker image sha256:52c6fc609bb75e38b52f60782235db5d34258fb753c501cf79cdfdc30735a9bf for [REDACTED]/base-images/utils/terraform:latest with digest [REDACTED]/base-images/utils/terraform@sha256:df81cf142dd1664a420269ee5ca69c090559be63104242e81a9d70089b95491a ...
section_end:1632764923:prepare_executor^Msection_start:1632764923:prepare_script^MPreparing environment
Running on runner-wzx5nsqv-project-216-concurrent-0 via hir-cloud1...
section_end:1632764924:prepare_script^Msection_start:1632764924:get_sources^MGetting source from Git repository
Fetching changes with git depth set to 50...
Reinitialized existing Git repository in /builds/cao/hir/earch/sandbox/max-tf-gl-be/.git/
Checking out 2bf658d9 as development...
Removing .terraform.lock.hcl
Removing .terraform/
Removing http-backend.tf
Removing plan.tfplan
Skipping Git submodules setup
section_end:1632764925:get_sources^Msection_start:1632764925:restore_cache^MRestoring cache
Checking cache for default-1...
No URL provided, cache will not be downloaded from shared cache server. Instead a local version of cache will be extracted.
Successfully extracted cache
section_end:1632764929:restore_cache^Msection_start:1632764929:download_artifacts^MDownloading artifacts
Downloading artifacts for plan_primary (72213)...
Downloading artifacts from coordinator... ok id=72213 responseStatus=200 OK token=sRSj32dk
section_end:1632764930:download_artifacts^Msection_start:1632764930:step_script^MExecuting "step_script" stage of the job script
Using docker image sha256:52c6fc609bb75e38b52f60782235db5d34258fb753c501cf79cdfdc30735a9bf for [REDACTED]/base-images/utils/terraform:latest with digest [REDACTED]/base-images/utils/terraform@sha256:df81cf142dd1664a420269ee5ca69c090559be63104242e81a9d70089b95491a ...
$ export GITLAB_TF_ADDRESS="${CI_API_V4_URL}/projects/${CI_PROJECT_ID}/terraform/state/${CI_ENVIRONMENT_SLUG}"
$ echo $GITLAB_TF_ADDRESS
https://[REDCATED]/api/v4/projects/216/terraform/state/primary-development
$ export TF_VAR_BUCKET_NAME="$TF_VAR_BUCKET_NAME-${CI_ENVIRONMENT_SLUG}"
$ echo -e "terraform { \n backend \"http\" {}\n}" >> http-backend.tf
$ terraform --version
Terraform v1.0.2
on linux_amd64
Your version of Terraform is out of date! The latest version
is 1.0.7. You can update by downloading from https://www.terraform.io/downloads.html
$ terraform init -reconfigure \ # collapsed multi-line command
Initializing the backend...
Successfully configured the backend "http"! Terraform will automatically
use this backend unless the backend configuration changes.
2021/09/27 17:48:51 [DEBUG] GET https://[REDCATED]/api/v4/projects/216/terraform/state/primary-development
Initializing provider plugins...
- Finding latest version of hashicorp/aws...
- Installing hashicorp/aws v3.60.0...
- Installed hashicorp/aws v3.60.0 (signed by HashiCorp)
Terraform has created a lock file .terraform.lock.hcl to record the provider
selections it made above. Include this file in your version control repository
so that Terraform can guarantee to make the same selections by default when
you run "terraform init" in the future.
Terraform has been successfully initialized!
You may now begin working with Terraform. Try running "terraform plan" to see
any changes that are required for your infrastructure. All Terraform commands
should now work.
If you ever set or change modules or backend configuration for Terraform,
rerun this command to reinitialize your working directory. If you forget, other
commands will detect it and remind you to do so if necessary.
$ terraform apply -lock=false -input=false $PLAN
2021/09/27 17:48:54 [DEBUG] GET https://[REDCATED]/api/v4/projects/216/terraform/state/primary-development
╷
│ Error: Error loading state: HTTP remote state endpoint requires auth
│
│
╵
section_end:1632764934:step_script^Msection_start:1632764934:cleanup_file_variables^MCleaning up file based variables
section_end:1632764934:cleanup_file_variables^MERROR: Job failed: exit code 1
Output of checks
Results of GitLab environment info
Expand for output related to GitLab environment info
(For installations with omnibus-gitlab package run and paste the output of: `sudo gitlab-rake gitlab:env:info`) (For installations from source run and paste the output of: `sudo -u git -H bundle exec rake gitlab:env:info RAILS_ENV=production`)
Results of GitLab application Check
Expand for output related to the GitLab application check
(For installations with omnibus-gitlab package run and paste the output of:
sudo gitlab-rake gitlab:check SANITIZE=true
)(For installations from source run and paste the output of:
sudo -u git -H bundle exec rake gitlab:check RAILS_ENV=production SANITIZE=true
)(we will only investigate if the tests are passing)