Terminal access fails on ACS and GKE
When trying to connect to an environment's terminal while on ACS, an error is generating saying "Connection failure".
In looking at the pod logs, it looks like there is some issue with the settings:
2017-05-03T16:54:49.914879664Z ==> /var/log/gitlab/gitlab-workhorse/current <==
2017-05-03T16:54:49.914883864Z 2017-05-03_16:54:49.91470 gitlab.azure.s1lv.com 10.244.1.8:48710 - - [2017-05-03 16:54:49.842157629 +0000 UTC] "GET /examples/minimal-ruby/environments/1/terminal HTTP/1.1" 200 18628 "https://gitlab.azure.s1lv.com/examples/minimal-ruby/environments" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_12_4) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/57.0.2987.133 Safari/537.36" 0.072499
2017-05-03T16:54:50.210811154Z
2017-05-03T16:54:50.210835555Z ==> /var/log/gitlab/gitlab-rails/production.log <==
2017-05-03T16:54:50.210840655Z Started GET "/health_check?token=SXBAQichEJasbtDSygrD" for 127.0.0.1 at 2017-05-03 16:54:50 +0000
2017-05-03T16:54:50.210843955Z Started GET "/health_check?token=SXBAQichEJasbtDSygrD" for 127.0.0.1 at 2017-05-03 16:54:50 +0000
2017-05-03T16:54:50.213177078Z Processing by HealthCheckController#index as HTML
2017-05-03T16:54:50.213196079Z Parameters: {"token"=>"SXBAQichEJasbtDSygrD"}
2017-05-03T16:54:50.213935186Z Processing by HealthCheckController#index as HTML
2017-05-03T16:54:50.214005887Z Parameters: {"token"=>"SXBAQichEJasbtDSygrD"}
2017-05-03T16:54:50.238745537Z Completed 200 OK in 24ms (Views: 0.2ms | ActiveRecord: 3.0ms)
2017-05-03T16:54:50.239548945Z
2017-05-03T16:54:50.239559345Z ==> /var/log/gitlab/gitlab-workhorse/current <==
2017-05-03T16:54:50.239562945Z 2017-05-03_16:54:50.23938 10.244.1.13:8005 10.244.1.1:44504 - - [2017-05-03 16:54:50.209644343 +0000 UTC] "GET /health_check?token=SXBAQichEJasbtDSygrD HTTP/1.1" 200 7 "" "Go-http-client/1.1" 0.029637
2017-05-03T16:54:50.239703646Z
2017-05-03T16:54:50.239711046Z ==> /var/log/gitlab/gitlab-rails/production.log <==
2017-05-03T16:54:50.239713746Z Completed 200 OK in 26ms (Views: 0.4ms | ActiveRecord: 5.8ms)
2017-05-03T16:54:50.323232290Z ==> /var/log/gitlab/gitlab-rails/production.log <==
2017-05-03T16:54:50.323237290Z Processing by Projects::EnvironmentsController#terminal_websocket_authorize as HTML
2017-05-03T16:54:50.323240390Z Parameters: {"namespace_id"=>"examples", "project_id"=>"minimal-ruby", "id"=>"1"}
2017-05-03T16:54:50.482196696Z
2017-05-03T16:54:50.482216897Z ==> /var/log/gitlab/sidekiq/current <==
2017-05-03T16:54:50.482238197Z 2017-05-03_16:54:50.48194 2017-05-03T16:54:50.481Z 660 TID-or00ptbok ReactiveCachingWorker JID-70a6cfcb6ee2a64c27c1b322 INFO: start
2017-05-03T16:54:50.485393529Z
2017-05-03T16:54:50.485408229Z ==> /var/log/gitlab/gitlab-rails/production.log <==
2017-05-03T16:54:50.485412429Z Completed 404 Not Found in 162ms (Views: 4.6ms | ActiveRecord: 15.0ms)
2017-05-03T16:54:50.486512140Z
2017-05-03T16:54:50.486523540Z ==> /var/log/gitlab/gitlab-workhorse/current <==
2017-05-03T16:54:50.486527640Z 2017-05-03_16:54:50.48635 gitlab.azure.s1lv.com 10.244.1.8:48724 - - [2017-05-03 16:54:50.279746851 +0000 UTC] "GET /examples/minimal-ruby/environments/1/terminal.ws HTTP/1.1" 404 9 "" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_12_4) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/57.0.2987.133 Safari/537.36" 0.206550
2017-05-03T16:54:50.488202857Z
2017-05-03T16:54:50.488213857Z ==> /var/log/gitlab/sidekiq/current <==
2017-05-03T16:54:50.488217957Z 2017-05-03_16:54:50.48799 2017-05-03T16:54:50.487Z 660 TID-or00ptbok ReactiveCachingWorker JID-70a6cfcb6ee2a64c27c1b322 INFO: fail: 0.007 sec
2017-05-03T16:54:50.488493560Z 2017-05-03_16:54:50.48826 2017-05-03T16:54:50.488Z 660 TID-or00ptbok WARN: {"context":"Job raised exception","job":{"class":"ReactiveCachingWorker","args":["KubernetesService",3],"retry":3,"queue":"reactive_caching","jid":"70a6cfcb6ee2a64c27c1b322","created_at":1493830490.4793725,"enqueued_at":1493830490.4800096,"error_message":"Incomplete settings","error_class":"RuntimeError","failed_at":1493830490.4876015,"retry_count":0},"jobstr":"{\"class\":\"ReactiveCachingWorker\",\"args\":[\"KubernetesService\",3],\"retry\":3,\"queue\":\"reactive_caching\",\"jid\":\"70a6cfcb6ee2a64c27c1b322\",\"created_at\":1493830490.4793725,\"enqueued_at\":1493830490.4800096}"}
2017-05-03T16:54:50.488521660Z 2017-05-03_16:54:50.48836 2017-05-03T16:54:50.488Z 660 TID-or00ptbok WARN: RuntimeError: Incomplete settings
2017-05-03T16:54:50.488747163Z 2017-05-03_16:54:50.48845 2017-05-03T16:54:50.488Z 660 TID-or00ptbok WARN: /opt/gitlab/embedded/service/gitlab-rails/app/models/project_services/kubernetes_service.rb:158:in `build_kubeclient!'
2017-05-03T16:54:50.488754663Z 2017-05-03_16:54:50.48846 /opt/gitlab/embedded/service/gitlab-rails/app/models/project_services/kubernetes_service.rb:123:in `calculate_reactive_cache'
2017-05-03T16:54:50.488758263Z 2017-05-03_16:54:50.48847 /opt/gitlab/embedded/service/gitlab-rails/app/models/concerns/reactive_caching.rb:80:in `block (3 levels) in exclusively_update_reactive_cache!'
2017-05-03T16:54:50.488761463Z 2017-05-03_16:54:50.48850 /opt/gitlab/embedded/service/gitlab-rails/app/models/concerns/reactive_caching.rb:113:in `enqueuing_update'
2017-05-03T16:54:50.488774763Z 2017-05-03_16:54:50.48850 /opt/gitlab/embedded/service/gitlab-rails/app/models/concerns/reactive_caching.rb:79:in `block (2 levels) in exclusively_update_reactive_cache!'
2017-05-03T16:54:50.488778063Z 2017-05-03_16:54:50.48850 /opt/gitlab/embedded/service/gitlab-rails/app/models/concerns/reactive_caching.rb:109:in `within_reactive_cache_lifetime'
2017-05-03T16:54:50.488780963Z 2017-05-03_16:54:50.48852 /opt/gitlab/embedded/service/gitlab-rails/app/models/concerns/reactive_caching.rb:78:in `block in exclusively_update_reactive_cache!'
2017-05-03T16:54:50.488790463Z 2017-05-03_16:54:50.48852 /opt/gitlab/embedded/service/gitlab-rails/app/models/concerns/reactive_caching.rb:103:in `locking_reactive_cache'
Accessing terminal.ws
also results in a 404, this log is from the ingress:
2017-05-03T17:31:36.835311872Z 73.165.8.176 - [73.165.8.176] - - [03/May/2017:17:31:36 +0000] "GET /examples/minimal-ruby/environments/1/terminal.ws HTTP/1.1" 404 9 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_12_4) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/57.0.2987.133 Safari/537.36" 811 0.037 10.244.1.13:8005 9 0.037 404