Setting environment in Gitlab CI crashes deployment of review app to k8s
Summary
I was trying to deploy an application to our staging k8s using kubectl and Helm version 3 (stack of application, if it matters - PHP-FPM + nginx, MariaDB, RabbitMQ, Redis, Mailhog). Deployment stage requires to add used Helm repositories and creating custom namespace, which has concatenated <project_name> and <branch_name>. It is checked whether given namespace exists, if not - it should create it. Provided environment variables (name and url) should allow to create custom environment for a review app. However, the stage crashes:
- It cannot get namespaces nor create them
- After manual creation of the namespace that will be used, provided ServiceAccount doesn't have access to list/create k8s objects
This is resolved by deleting environment part, but still, it doesn't let the pipeline work as expected and needed.
Steps to reproduce
- Attach k8s cluster to your GitLab project along with any permissions that it needs (my SA inherits from cluster-admin, thus having privileges to do anything basically)
- Create a simple Gitlab CI pipeline, using any kubectl command that operates on objects (kubectl create ns <namespace_name> in given example)
- Add environment part - name and url.
- Run the pipeline by pushing changes to Gitlab
What is the current bug behavior?
$ kubectl get ns $NAMESPACE || kubectl create ns $NAMESPACE
Error from server (Forbidden): namespaces "xxx-update-ci" is forbidden: User "system:serviceaccount:kube-system:gitlab-admin" cannot get resource "namespaces" in API group "" in the namespace "xxxx-update-ci"
Error from server (Forbidden): namespaces is forbidden: User "system:serviceaccount:kube-system:gitlab-admin" cannot create resource "namespaces" in API group "" at the cluster scope
NAMESPACE is defined earlier:
$ export NAMESPACE="$CI_PROJECT_NAMESPACE-$CI_COMMIT_REF_SLUG"
What is the expected correct behavior?
The line stated above should get namespace defined, or - if it doesn't exist - create one.
Results of GitLab environment info
Expand for output related to GitLab environment info
System information System: Current User: git Using RVM: no Ruby Version: 2.6.3p62 Gem Version: 2.7.9 Bundler Version:1.17.3 Rake Version: 12.3.3 Redis Version: 3.2.12 Git Version: 2.24.1 Sidekiq Version:5.2.7 Go Version: unknown GitLab information Version: 12.6.4 Revision: 70900054dfe Directory: /opt/gitlab/embedded/service/gitlab-rails DB Adapter: PostgreSQL DB Version: 10.9 URL: https://git.xxx.com HTTP Clone URL: https://git.xxx.com/some-group/some-project.git SSH Clone URL: git@git.xxx.com:some-group/some-project.git Using LDAP: no Using Omniauth: yes Omniauth Providers: google_oauth2 GitLab Shell Version: 10.3.0 Repository storage paths: - default: /var/opt/gitlab/git-data/repositories GitLab Shell path: /opt/gitlab/embedded/service/gitlab-shell Git: /opt/gitlab/embedded/bin/git
Results of GitLab application Check
Expand for output related to the GitLab application check
Checking GitLab subtasks ... Checking GitLab Shell ... GitLab Shell: ... GitLab Shell version >= 10.3.0 ? ... OK (10.3.0) Running /opt/gitlab/embedded/service/gitlab-shell/bin/check Internal API available: OK Redis available via internal API: OK gitlab-shell self-check successful Checking GitLab Shell ... Finished Checking Gitaly ... Gitaly: ... default ... OK Checking Gitaly ... Finished Checking Sidekiq ... Sidekiq: ... Running? ... yes Number of Sidekiq processes ... 1 Checking Sidekiq ... Finished Checking Incoming Email ... Incoming Email: ... Reply by email is disabled in config/gitlab.yml Checking Incoming Email ... Finished Checking LDAP ... LDAP: ... LDAP is disabled in config/gitlab.yml Checking LDAP ... Finished Checking GitLab App ... Git configured correctly? ... yes Database config exists? ... yes All migrations up? ... yes Database contains orphaned GroupMembers? ... no GitLab config exists? ... yes GitLab config up to date? ... yes Log directory writable? ... yes Tmp directory writable? ... yes Uploads directory exists? ... yes Uploads directory has correct permissions? ... yes Uploads directory tmp has correct permissions? ... no Try fixing it: sudo chown -R git /var/opt/gitlab/gitlab-rails/uploads sudo find /var/opt/gitlab/gitlab-rails/uploads -type f -exec chmod 0644 {} \; sudo find /var/opt/gitlab/gitlab-rails/uploads -type d -not -path /var/opt/gitlab/gitlab-rails/uploads -exec chmod 0700 {} \; For more information see: doc/install/installation.md in section "GitLab" Please fix the error above and rerun the checks. Init script exists? ... skipped (omnibus-gitlab has no init script) Init script up-to-date? ... skipped (omnibus-gitlab has no init script) Projects have namespace: ... 4/1 ... yes 51/4 ... yes 51/5 ... yes 9/6 ... yes 9/7 ... yes 14/8 ... yes 14/9 ... yes 14/10 ... yes 38/13 ... yes 38/14 ... yes 40/15 ... yes 33/16 ... yes 9/17 ... yes 14/18 ... yes 33/25 ... yes 14/27 ... yes 14/28 ... yes 7/29 ... yes 7/30 ... yes 7/31 ... yes 7/32 ... yes 40/33 ... yes 14/35 ... yes 48/36 ... yes 48/37 ... yes 48/38 ... yes 7/39 ... yes 49/42 ... yes 49/43 ... yes 49/44 ... yes 51/45 ... yes 51/46 ... yes 51/47 ... yes 51/48 ... yes 33/49 ... yes 52/51 ... yes 52/52 ... yes 49/53 ... yes 54/54 ... yes 54/58 ... yes 40/59 ... yes 4/61 ... yes 55/62 ... yes 55/63 ... yes 4/64 ... yes 4/65 ... yes 4/66 ... yes 57/67 ... yes 56/68 ... yes 4/69 ... yes 58/70 ... yes 58/71 ... yes 14/72 ... yes 59/73 ... yes 51/74 ... yes 51/75 ... yes 51/76 ... yes 4/77 ... yes 40/78 ... yes 33/81 ... yes 51/82 ... yes 51/83 ... yes 51/84 ... yes 51/85 ... yes 51/86 ... yes 13/87 ... yes 54/88 ... yes 60/89 ... yes 64/93 ... yes 69/94 ... yes 22/95 ... yes 71/97 ... yes 71/98 ... yes 73/102 ... yes 75/103 ... yes 73/104 ... yes 73/105 ... yes 73/106 ... yes 76/109 ... yes 73/110 ... yes 117/112 ... yes 73/113 ... yes 75/114 ... yes 73/115 ... yes 73/117 ... yes 73/118 ... yes 73/119 ... yes 73/120 ... yes 54/121 ... yes 85/122 ... yes 73/123 ... yes 33/124 ... yes 73/125 ... yes 73/126 ... yes 73/127 ... yes 65/131 ... yes 64/132 ... yes 71/133 ... yes 100/134 ... yes 65/135 ... yes 110/136 ... yes 94/137 ... yes 111/138 ... yes 112/140 ... yes 228/141 ... yes 73/142 ... yes 113/143 ... yes 71/144 ... yes 117/145 ... yes 123/146 ... yes 126/148 ... yes 67/150 ... yes 67/151 ... yes 54/152 ... yes 133/155 ... yes 130/156 ... yes 136/157 ... yes 97/158 ... yes 142/159 ... yes 33/164 ... yes 139/166 ... yes 126/167 ... yes 153/168 ... yes 126/169 ... yes 126/170 ... yes 154/171 ... yes 97/172 ... yes 60/173 ... yes 126/174 ... yes 126/175 ... yes 154/176 ... yes 126/177 ... yes 162/179 ... yes 164/180 ... yes 122/181 ... yes 166/182 ... yes 166/183 ... yes 154/185 ... yes 139/186 ... yes 8/188 ... yes 154/189 ... yes 65/193 ... yes 162/195 ... yes 162/197 ... yes 162/198 ... yes 162/199 ... yes 60/200 ... yes 176/203 ... yes 213/204 ... yes 162/205 ... yes 162/207 ... yes 139/209 ... yes 139/211 ... yes 139/212 ... yes 162/213 ... yes 228/215 ... yes 162/284 ... yes 139/285 ... yes 179/286 ... yes 188/287 ... yes 188/288 ... yes 187/289 ... yes 180/290 ... yes 188/291 ... yes 188/292 ... yes 139/293 ... yes 179/295 ... yes 190/296 ... yes 190/298 ... yes 188/299 ... yes 162/301 ... yes 36/303 ... yes 188/304 ... yes 73/305 ... yes 182/306 ... yes 166/307 ... yes 188/308 ... yes 186/309 ... yes 188/310 ... yes 139/314 ... yes 188/315 ... yes 188/316 ... yes 188/317 ... yes 197/318 ... yes 188/319 ... yes 188/320 ... yes 73/321 ... yes 182/322 ... yes 188/323 ... yes 97/324 ... yes 176/327 ... yes 46/328 ... yes 139/329 ... yes 73/331 ... yes 203/332 ... yes 97/333 ... yes 201/334 ... yes 218/335 ... yes 162/336 ... yes 65/337 ... yes 210/338 ... yes 73/339 ... yes 203/340 ... yes 210/341 ... yes 176/342 ... yes 218/343 ... yes 213/344 ... yes 214/345 ... yes 214/346 ... yes 65/347 ... yes 147/348 ... yes 136/349 ... yes 162/350 ... yes 218/351 ... yes 162/353 ... yes 227/355 ... yes 243/356 ... yes 176/357 ... yes 97/358 ... yes 97/359 ... yes 60/360 ... yes 225/363 ... yes 225/364 ... yes 225/365 ... yes 225/366 ... yes 130/367 ... yes 162/369 ... yes 176/370 ... yes 216/371 ... yes 227/372 ... yes 218/373 ... yes 232/376 ... yes 214/378 ... yes 222/379 ... yes 174/380 ... yes 139/381 ... yes 65/383 ... yes 214/385 ... yes 210/386 ... yes 210/387 ... yes 231/388 ... yes 243/389 ... yes 214/391 ... yes 214/392 ... yes 234/393 ... yes 237/394 ... yes 230/395 ... yes 234/396 ... yes 4/397 ... yes 234/398 ... yes 234/399 ... yes 214/400 ... yes 210/401 ... yes 210/402 ... yes 237/403 ... yes 241/405 ... yes 218/407 ... yes 218/408 ... yes 218/409 ... yes 176/410 ... yes 257/411 ... yes 235/412 ... yes 257/413 ... yes 260/414 ... yes 257/418 ... yes 257/419 ... yes 245/421 ... yes 261/423 ... yes 248/425 ... yes 257/426 ... yes 250/427 ... yes 248/428 ... yes 260/429 ... yes 260/430 ... yes 218/431 ... yes 268/432 ... yes 268/433 ... yes 268/434 ... yes 242/435 ... yes 260/436 ... yes 210/437 ... yes 210/438 ... yes 260/439 ... yes 269/440 ... yes 253/441 ... yes 130/442 ... yes 210/443 ... yes 218/444 ... yes 64/445 ... yes 246/446 ... yes 260/447 ... yes 248/448 ... yes 214/449 ... yes 214/450 ... yes 269/451 ... yes 269/452 ... yes 260/453 ... yes 245/454 ... yes 210/455 ... yes 241/457 ... yes 64/458 ... yes 271/460 ... yes 259/461 ... yes 271/463 ... yes 271/464 ... yes 271/465 ... yes 13/466 ... yes 257/467 ... yes 269/468 ... yes 73/469 ... yes 257/470 ... yes 4/471 ... yes 257/472 ... yes 271/473 ... yes 234/474 ... yes 257/475 ... yes 257/476 ... yes 257/477 ... yes 13/478 ... yes 210/480 ... yes 73/481 ... yes 257/482 ... yes 230/483 ... yes 278/484 ... yes 257/485 ... yes 130/487 ... yes 64/488 ... yes 257/489 ... yes 73/490 ... yes 271/491 ... yes 257/492 ... yes 218/493 ... yes 130/494 ... yes 218/495 ... yes 174/496 ... yes 256/497 ... yes 271/498 ... yes 64/499 ... yes 218/501 ... yes 271/502 ... yes 225/503 ... yes 203/504 ... yes 271/506 ... yes 162/507 ... yes 64/508 ... yes 287/509 ... yes 13/510 ... yes 64/511 ... yes 288/512 ... yes 287/513 ... yes 218/514 ... yes 290/515 ... yes 288/516 ... yes 261/517 ... yes Redis version >= 2.8.0? ... yes Ruby version >= 2.5.3 ? ... yes (2.6.3) Git version >= 2.22.0 ? ... yes (2.24.1) Git user has default SSH configuration? ... yes Active users: ... 106 Is authorized keys file accessible? ... yes Checking GitLab App ... Finished Checking GitLab subtasks ... Finished
Possible fixes
Remove environment variables - it is impossible to run a review app this way, but it lets the stage complete.
It may be linked to https://gitlab.com/gitlab-com/support-forum/issues/4497, however GitLab-managed cluster is not ticked in settings.