gitlab-terraform: command not found
Summary
When trying to run the Gitlab Terraform EKS pipeline to deploy the cluster as per these instructions https://docs.gitlab.com/ee/user/infrastructure/clusters/connect/new_eks_cluster.html, the pipeline is failing with the error "gitlab-terraform: command not found".
Steps to reproduce
-
INSTALL GITLAB RUNNER ON THE GITLAB VM: [ec2-user@ip-10-0-0-221 ~]$ fips-mode-setup --check FIPS mode is enabled.
curl -L "https://packages.gitlab.com/install/repositories/runner/gitlab-runner/script.rpm.sh" | sudo bash
sudo yum install gitlab-runner
-
GENERATE REGISTRATION TOKEN AND REGISTER GITLAB RUNNER: Gitlab > Projects > select project > Settings > CI/CD > Runners > New Project Runner > Operating Systems: Linux > Tags: xiohelm-cicd > Submit.
Execute the instructions from the Register Runner page of the UI on the GitLab VM: gitlab-runner register --url https://gitlab.xxx.com --token xxxxx Enter the GitLab instance URL: https://gitlab.xxx.com Enter a name for the runner: gitlab.xxx.com Enter an executor: shell
- IMPORT THE EXAMPLE PROJECT INTO GITLAB: Enable imports into Gitlab: Gitlab > Admin area > Settings > General > Visibility and Access Controls > Import Sources: GitHub, Repository by URL, GitLab export > Save Changes.
Import example project into GitLab: GitLab > Projects > View All Projects > New Project > Import Project > Repository by URL > URL: https://gitlab.com/gitlab-org/configure/examples/gitlab-terraform-eks.git > Project URL: https://gitlab.xxxx.com/xiohelm-cicd/gitlab-terraform-eks > Name: Gitlab Terraform Eks > Create Project.
- REGISTER THE GITLAB KUBERNETES AGENT: Create the Gitlab Agent for Kubernetes: Gitlab > Projects > Gitlab Terraform Eks > Infrastructure (left menu) > Kubernetes cluster > Connect a cluster > Select an agent: eks-agent > Register.
Agent Access Token: xxx
- SET UP AWS CREDENTIALS: AWS > IAM > Users > Create User > User name: gitlab-cicd > Provide user access to the AWS Management Console: select > Console Password: Custom Password: Meow!Meow!_#123 > Users must create a new password at next sign-in: deselect > Next > Attach policies directly > Create Policy > In the new tab that opens for creating the policy, select JSON and paste in the following policy overwriting any content in the Policy Editor:
{ "Version": "2012-10-17", "Statement": [ { "Sid": "VisualEditor0", "Effect": "Allow", "Action": [ "ec2:", "eks:", "elasticloadbalancing:", "autoscaling:", "cloudwatch:", "logs:", "kms:DescribeKey", "iam:AddRoleToInstanceProfile", "iam:AttachRolePolicy", "iam:CreateInstanceProfile", "iam:CreateRole", "iam:CreateServiceLinkedRole", "iam:GetRole", "iam:ListAttachedRolePolicies", "iam:ListRolePolicies", "iam:ListRoles", "iam:PassRole", "iam:DetachRolePolicy", "iam:ListInstanceProfilesForRole", "iam:DeleteRole" ], "Resource": "*" } ] }
Next > Policy name: gitlab-cicd > Create Policy > Back in the IAM User tab, refresh the permission policies and select the gitlab-cicd policy > Next > Create User > Return to User List > Select the gitlab-cicd user > Security Credentials tab > Create Access Key > Other > Next > Create Access Key > Download .csv file > Done.
-
SET THE GITLAB VARIABLES: Gitlab > Groups > select the group cicd (must have owner permissions to the group) > Settings (left menu) > CICD > Variables > Add Variables > add the following variables with credentials from the gitlab-cicd user: AWS_ACCESS_KEY_ID AWS_SECRET_ACCESS_KEY AWS_DEFAULT_REGION TF_VAR_agent_token TF_VAR_kas_address
-
CONFIGURE RUNNER TO RUN UNTAGGED JOBS: Gitlab > Groups > cicd > Runners > Edit the runner > select Run untagged jobs > Save Changes.
-
DEPLOY CLUSTER VIA PIPELINE:
If jobs have pending status, and clicking on it shows that the job is pending due to no runners available to the project, click the runners hyperlink and enable the gitlab.teloscloud.com runner for this project.
gitlab-runner run
-
I ALSO TRIED THIS METHOD OF REGISTERING, IN CASE THIS WORKED BETTER, BUT IT DID NOT: The following steps are from the instructions at Gitlab > Project > Settings > CICD > Runners > Project Runners elipses > Show Runner Installation and Registration Instructions (These instructions include the registration command, so the registration step, which is next, can be skipped):
sudo curl -L --output /usr/local/bin/gitlab-runner https://gitlab-runner-downloads.s3.amazonaws.com/latest/binaries/gitlab-runner-linux-amd64
sudo chmod +x /usr/local/bin/gitlab-runner
sudo useradd --comment 'GitLab Runner' --create-home gitlab-runner --shell /bin/bash
sudo /usr/local/bin/gitlab-runner install --user=gitlab-runner --working-directory=/home/gitlab-runner
gitlab-runner register --url https://gitlab.XXX.com/ --registration-token xxxx Enter the GitLab instance URL: https://gitlab.xxx.com Enter the registration token: [xxxxx]: xxxxx Enter a description for the runner: standalone Enter tags for the runner (comma-separated): xiohelm-cicd Enter optional maintenance note for the runner: Enter an executor: shell
REFERENCES: https://docs.gitlab.com/16.0/ee/user/infrastructure/clusters/connect/new_eks_cluster.html https://docs.gitlab.com/16.0/runner/install/linux-repository.html https://docs.gitlab.com/16.0/ee/ci/runners/register_runner.html https://docs.gitlab.com/16.0/ee/user/infrastructure/clusters/connect/new_eks_cluster.html
What is the current bug behavior?
validate job fails as follows:
Using Shell (bash) executor... Preparing environment 00:00 Running on ip-10-0-0-221.us-gov-east-1.compute.internal... Getting source from Git repository 00:01 Fetching changes with git depth set to 20... Reinitialized existing Git repository in /home/ec2-user/builds/xvXF_QehX/0/xiohelm-cicd/gitlab-terraform-eks/.git/ Checking out 75e21b71 as detached HEAD (ref is main)... Skipping Git submodules setup Restoring cache 00:00 Checking cache for /home/ec2-user/builds/xvXF_QehX/0/xiohelm-cicd/gitlab-terraform-eks-1-protected... Runtime platform arch=amd64 os=linux pid=17851 revision=79704081 version=16.0.1 No URL provided, cache will not be downloaded from shared cache server. Instead a local version of cache will be extracted. Successfully extracted cache Executing "step_script" stage of the job script 00:00 $ gitlab-terraform validate bash: line 139: gitlab-terraform: command not found Cleaning up project directory and file based variables 00:00 ERROR: Job failed: exit status 1
What is the expected correct behavior?
fmt and validate jobs should execute successfully to deploy the cluster. I saw some issues on stack overflow where people had this same error and had to add in the extends lines like "extends: .terraform:fmt
"
but it seems that these are already present in the latest release of https://gitlab.com/gitlab-org/gitlab/-/blob/master/lib/gitlab/ci/templates/Terraform.gitlab-ci.yml that @timofurrer fixed, so not sure why it still isn't working. @timofurrer @alberts-gitlab @vyaklushin is there anything else I need to do to utilize this? also, I saw people on stack overflow mentioning that they needed to point to the latest hashicorp/terraform image, and I tried adding this section into the gitlab-ci.yaml file, but it said it conflicted with an image already defined upstream:
default: image: name: hashicorp/terraform:latest entrypoint: - /usr/bin/env - "PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin"
Results of GitLab environment info
sudo gitlab-rake gitlab:env:info
System information System: Proxy: no Current User: git Using RVM: no Ruby Version: 3.0.6p216 Gem Version: 3.4.13 Bundler Version:2.4.13 Rake Version: 13.0.6 Redis Version: 6.2.11 Sidekiq Version:6.5.7 Go Version: go1.19.9 linux/amd64
GitLab information Version: 16.0.1-ee Revision: 29e1314f Directory: /opt/gitlab/embedded/service/gitlab-rails DB Adapter: PostgreSQL DB Version: 13.8 URL: https://gitlab.xxx.com HTTP Clone URL: https://gitlab.xxx.com/some-group/some-project.git SSH Clone URL: git@gitlab.xxx.com:some-group/some-project.git Elasticsearch: no Geo: no Using LDAP: no Using Omniauth: yes Omniauth Providers:
GitLab Shell Version: 14.20.0 Repository storages:
- default: unix:/var/opt/gitlab/gitaly/gitaly.socket
Results of GitLab application Check
sudo gitlab-rake gitlab:check SANITIZE=true
Checking GitLab subtasks ...
Checking GitLab Shell ...
GitLab Shell: ... GitLab Shell version >= 14.20.0 ? ... OK (14.20.0) Running /opt/gitlab/embedded/service/gitlab-shell/bin/check Internal API available: OK Redis available via internal API: OK gitlab-shell self-check successful
Checking GitLab Shell ... Finished
Checking Gitaly ...
Gitaly: ... default ... OK
Checking Gitaly ... Finished
Checking Sidekiq ...
Sidekiq: ... Running? ... yes Number of Sidekiq processes (cluster/worker) ... 1/1
Checking Sidekiq ... Finished
Checking Incoming Email ...
Incoming Email: ... Reply by email is disabled in config/gitlab.yml
Checking Incoming Email ... Finished
Checking LDAP ...
LDAP: ... LDAP is disabled in config/gitlab.yml
Checking LDAP ... Finished
Checking GitLab App ...
Database config exists? ... yes All migrations up? ... yes Database contains orphaned GroupMembers? ... no GitLab config exists? ... yes GitLab config up to date? ... yes Cable config exists? ... yes Resque config exists? ... yes Log directory writable? ... yes Tmp directory writable? ... yes Uploads directory exists? ... yes Uploads directory has correct permissions? ... yes Uploads directory tmp has correct permissions? ... yes Systemd unit files or init script exist? ... skipped (omnibus-gitlab has neither init script nor systemd units) Systemd unit files or init script up-to-date? ... skipped (omnibus-gitlab has neither init script nor systemd units) Projects have namespace: ... 8/2 ... yes 8/3 ... yes Redis version >= 6.0.0? ... yes Ruby version >= 2.7.2 ? ... yes (3.0.6) Git user has default SSH configuration? ... yes Active users: ... 4 Is authorized keys file accessible? ... yes GitLab configured to store new projects in hashed storage? ... yes All projects are in hashed storage? ... yes Elasticsearch version 7.x-8.x or OpenSearch version 1.x ... skipped (Advanced Search is disabled) All migrations must be finished before doing a major upgrade ... skipped (Advanced Search is disabled)
Checking GitLab App ... Finished
Checking GitLab subtasks ... Finished