[Investigate]GitLab Managed Terraform State fails auth with CI_JOB_TOKEN

Summary

After upgrading to GitLab version 14.1.2, using ${CI_JOB_TOKEN} fails authorization in a CI pipeline job. This was submitted on behalf of a US Federal customer (internal link only), so information has been redacted from the logs.

Steps to reproduce

Create a job that uses ${CI_JOB_TOKEN} to authorized with the Terraform state backend

Example Project

An example project wasn't created, however, here is the .gitlab-ci.yml file contents:

workflow:
  rules:
    - if: $CI_COMMIT_BRANCH

image:
  name: hashicorp/terraform:light
  entrypoint:
    - '/usr/bin/env'
    - 'PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin'

# Default output file for Terraform plan
variables:
  PLAN: plan.tfplan

cache:
  paths:
    - .terraform

before_script:
    # Set up state file per environment
  - export GITLAB_TF_ADDRESS="${CI_API_V4_URL}/projects/${CI_PROJECT_ID}/terraform/state/${CI_ENVIRONMENT_SLUG}"
  - echo $GITLAB_TF_ADDRESS
    # Append bucket name with the environment name
  - export TF_VAR_BUCKET_NAME="$TF_VAR_BUCKET_NAME-${CI_ENVIRONMENT_SLUG}"
    # Write terraform http backend definition to a file as it cannot be a var
  - echo -e "terraform { \n  backend \"http\" {}\n}" >> http-backend.tf
  - terraform --version
  - |
    terraform init -reconfigure \
    -backend-config="username=gitlab-ci-token" \
    -backend-config="password=${CI_JOB_TOKEN}" \
    -backend-config="address=${GITLAB_TF_ADDRESS}" \
    -backend-config="lock_address=${GITLAB_TF_ADDRESS}/lock" \
    -backend-config="unlock_address=${GITLAB_TF_ADDRESS}/unlock"

stages:
  - validate
  - plan
  - apply
  - destroy

validate:
  stage: validate
  script: terraform validate

.plan_tpl:
  stage: plan
  dependencies: [validate]
  script: terraform plan -lock=false -out=$PLAN
  rules:
    - if: $CI_COMMIT_BRANCH == 'development'
      when: on_success
    - if: $CI_COMMIT_BRANCH == 'staging'
      when: on_success
    - if: $CI_COMMIT_BRANCH == 'production'
      when: on_success
    - when: never
  artifacts:
    name: plan
    paths:
      - $PLAN

.apply_tpl:
  stage: apply
  script: terraform apply -lock=false -input=false $PLAN
  rules:
    - if: $CI_COMMIT_BRANCH == 'development'
      when: manual
    - if: $CI_COMMIT_BRANCH == 'staging'
      when: manual
    - if: $CI_COMMIT_BRANCH == 'production'
      when: manual
    - when: never

.destroy_tpl:
  stage: destroy
  script: terraform destroy -lock=false -auto-approve
  rules:
    - if: $CI_COMMIT_BRANCH == 'development' && $DESTROY_JOBS_ENABLED
      when: manual
    - if: $CI_COMMIT_BRANCH == 'staging' && $DESTROY_JOBS_ENABLED
      when: manual
    - if: $CI_COMMIT_BRANCH == 'production' && $DESTROY_JOBS_ENABLED
      when: manual
    - when: never

plan_primary:
  extends: [ .plan_tpl]
  environment:
    name: primary-${CI_COMMIT_BRANCH}
    action: prepare

apply_primary:
  extends: [ .apply_tpl ]
  environment:
    name: primary-${CI_COMMIT_BRANCH}
    on_stop: destroy_primary
  needs: [plan_primary]

destroy_primary:
  extends: [ .destroy_tpl  ]
  environment:
    name: primary-${CI_COMMIT_BRANCH}
    action: stop
  needs: [apply_primary, plan_primary]

plan_secondary:
  extends: [ .plan_tpl]
  environment:
    name: secondary-${CI_COMMIT_BRANCH}
    action: prepare

apply_secondary:
  extends: [ .apply_tpl ]
  environment:
    name: secondary-${CI_COMMIT_BRANCH}
    on_stop: destroy_secondary
  needs: [plan_secondary]

destroy_secondary:
  extends: [ .destroy_tpl ]
  environment:
    name: secondary-${CI_COMMIT_BRANCH}
    action: stop
  needs: [apply_secondary, plan_secondary]

What is the current bug behavior?

Using ${CI_JOB_TOKEN} to authenticate to the terraform managed state backend fails, however, using a personal access token and project access token authenticated to the terraform managed state backend successfully.

What is the expected correct behavior?

The CI job should be able to authorized with the Terraform state backend

Relevant logs and/or screenshots

Running with gitlab-runner 14.1.0 (8925d9a0)
  on hir-earch-group Wzx5nsQV
section_start:1632764922:resolve_secrets^MResolving secrets
section_end:1632764922:resolve_secrets^Msection_start:1632764922:prepare_executor^MPreparing the "docker" executor
Using Docker executor with image [REDACTED]/base-images/utils/terraform:latest ...
Pulling docker image [REDACTED]/base-images/utils/terraform:latest ...
Using docker image sha256:52c6fc609bb75e38b52f60782235db5d34258fb753c501cf79cdfdc30735a9bf for [REDACTED]/base-images/utils/terraform:latest with digest [REDACTED]/base-images/utils/terraform@sha256:df81cf142dd1664a420269ee5ca69c090559be63104242e81a9d70089b95491a ...
section_end:1632764923:prepare_executor^Msection_start:1632764923:prepare_script^MPreparing environment
Running on runner-wzx5nsqv-project-216-concurrent-0 via hir-cloud1...
section_end:1632764924:prepare_script^Msection_start:1632764924:get_sources^MGetting source from Git repository
Fetching changes with git depth set to 50...
Reinitialized existing Git repository in /builds/cao/hir/earch/sandbox/max-tf-gl-be/.git/
Checking out 2bf658d9 as development...
Removing .terraform.lock.hcl
Removing .terraform/
Removing http-backend.tf
Removing plan.tfplan

Skipping Git submodules setup
section_end:1632764925:get_sources^Msection_start:1632764925:restore_cache^MRestoring cache
Checking cache for default-1...
No URL provided, cache will not be downloaded from shared cache server. Instead a local version of cache will be extracted.
Successfully extracted cache
section_end:1632764929:restore_cache^Msection_start:1632764929:download_artifacts^MDownloading artifacts
Downloading artifacts for plan_primary (72213)...
Downloading artifacts from coordinator... ok        id=72213 responseStatus=200 OK token=sRSj32dk
section_end:1632764930:download_artifacts^Msection_start:1632764930:step_script^MExecuting "step_script" stage of the job script
Using docker image sha256:52c6fc609bb75e38b52f60782235db5d34258fb753c501cf79cdfdc30735a9bf for [REDACTED]/base-images/utils/terraform:latest with digest [REDACTED]/base-images/utils/terraform@sha256:df81cf142dd1664a420269ee5ca69c090559be63104242e81a9d70089b95491a ...
$ export GITLAB_TF_ADDRESS="${CI_API_V4_URL}/projects/${CI_PROJECT_ID}/terraform/state/${CI_ENVIRONMENT_SLUG}"
$ echo $GITLAB_TF_ADDRESS
https://[REDCATED]/api/v4/projects/216/terraform/state/primary-development
$ export TF_VAR_BUCKET_NAME="$TF_VAR_BUCKET_NAME-${CI_ENVIRONMENT_SLUG}"
$ echo -e "terraform { \n  backend \"http\" {}\n}" >> http-backend.tf
$ terraform --version
Terraform v1.0.2
on linux_amd64

Your version of Terraform is out of date! The latest version
is 1.0.7. You can update by downloading from https://www.terraform.io/downloads.html
$ terraform init -reconfigure \ # collapsed multi-line command

Initializing the backend...

Successfully configured the backend "http"! Terraform will automatically
use this backend unless the backend configuration changes.
2021/09/27 17:48:51 [DEBUG] GET https://[REDCATED]/api/v4/projects/216/terraform/state/primary-development

Initializing provider plugins...
- Finding latest version of hashicorp/aws...
- Installing hashicorp/aws v3.60.0...
- Installed hashicorp/aws v3.60.0 (signed by HashiCorp)

Terraform has created a lock file .terraform.lock.hcl to record the provider
selections it made above. Include this file in your version control repository
so that Terraform can guarantee to make the same selections by default when
you run "terraform init" in the future.

Terraform has been successfully initialized!

You may now begin working with Terraform. Try running "terraform plan" to see
any changes that are required for your infrastructure. All Terraform commands
should now work.

If you ever set or change modules or backend configuration for Terraform,
rerun this command to reinitialize your working directory. If you forget, other
commands will detect it and remind you to do so if necessary.
$ terraform apply -lock=false -input=false $PLAN
2021/09/27 17:48:54 [DEBUG] GET https://[REDCATED]/api/v4/projects/216/terraform/state/primary-development
╷
│ Error: Error loading state: HTTP remote state endpoint requires auth
│
│
╵
section_end:1632764934:step_script^Msection_start:1632764934:cleanup_file_variables^MCleaning up file based variables
section_end:1632764934:cleanup_file_variables^MERROR: Job failed: exit code 1

Output of checks

Results of GitLab environment info

Expand for output related to GitLab environment info

(For installations with omnibus-gitlab package run and paste the output of:
`sudo gitlab-rake gitlab:env:info`)

(For installations from source run and paste the output of:
`sudo -u git -H bundle exec rake gitlab:env:info RAILS_ENV=production`)

Results of GitLab application Check

Expand for output related to the GitLab application check

(For installations with omnibus-gitlab package run and paste the output of:
sudo gitlab-rake gitlab:check SANITIZE=true)
(For installations from source run and paste the output of:
sudo -u git -H bundle exec rake gitlab:check RAILS_ENV=production SANITIZE=true)
(we will only investigate if the tests are passing)

Possible fixes

Edited Jun 16, 2022 by Nicolò Maria Mezzopera