### Summary I'm unable to connect to *web terminal* for pod environments. Note: NOT runners debug. On a private kubernetes cluster. ### Steps to reproduce * Autodevops deploy a pod/project to the cluster * Deployed project -> Operations -> Environments -> Click Terminal ### What is the current *bug* behavior? The web terminal always displays "Connection failure". Request on terminal.ws returns a 200. ### What is the expected *correct* behavior? Direct access to nodes by k8 exec ### Relevant logs and/or screenshots Connection to K8 cluster works fine (Autodevops, deploy, review etc). The deployed pod is correctly annotated and accessible (ingress). ``` metadata: annotations: app.gitlab.com/app: group-name-project-name app.gitlab.com/env: staging ``` On the terminal page, the browser console shows; ``` terminal.js:49 WebSocket connection to 'wss://gitlab.mysite.com/group-name/project-name/-/environments/32/terminal.ws' failed: Error during WebSocket handshake: Unexpected response code: 200 ``` That it seems to me particularly wrong as I was expecting a *101 Switching Protocols* or a 404/5xx at least. **Note: I have gitlab behind an Apache 2.4.25-3+deb9u9 reverse proxy in the middle (forwarding to gitlab-ngnix) with this configuration for the proxy headers:** ``` ProxyRequests off ProxyPreserveHost on RewriteEngine on RewriteCond %{HTTP:Upgrade} websocket [NC] RewriteCond %{HTTP:Connection} upgrade [NC] RewriteRule .* "wss:/gitlab.mysite.com/$1" [P,L] ProxyPass / http://gitlab.mysite.com/ ProxyPassReverse / http://gitlab.mysite.com/ ``` Do I need a different configuration? Can you give any example of it ? *Request* ``` Request URL: wss://gitlab.mysite.com/mysite/project-name/-/environments/32/terminal.ws Connection: Upgrade Host: gitlab.mysite.com Origin: https://gitlab.mysite.com Sec-WebSocket-Extensions: permessage-deflate; client_max_window_bits Sec-WebSocket-Key: xxxxxxxxxxxxxxxxxxxxx Sec-WebSocket-Protocol: terminal.gitlab.com Sec-WebSocket-Version: 13 Upgrade: websocket GET /group-name/project-name/-/environments/32/terminal HTTP/1.1 Referer: https://gitlab.mysite.com/group-name/project-name/-/environments ``` *Response* ``` Request URL: https://gitlab.mysite.com/group-name/project-name/-/environments/32/terminal Request Method: GET Status Code: 200 OK Remote Address: xx.xx.xxx.xxx:443 Server: nginx ``` *gitlab-ctl tail gitlab-workhorse* ``` {"correlation_id":"pOzqjZekova","duration_ms":0,"host":"gitlab.mysite.com","level":"info","method":"GET","msg":"access","proto":"HTTP/1.1","referrer":"https://gitlab.mysite.com/group-name/project-name/-/environments/35/terminal","remote_addr":"127.0.0.1:0","remote_ip":"127.0.0.1","status":200,"system":"http","time":"2020-02-06T09:55:08+01:00","uri":"/assets/webpack/vendors~select2.c6af6b5d.chunk.js","user_agent":"Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/79.0.3945.130 Safari/537.36","written_bytes":17928} {"correlation_id":"3gQFg3D0c2a","duration_ms":295,"host":"gitlab.mysite.com","level":"info","method":"GET","msg":"access","proto":"HTTP/1.1","referrer":"","remote_addr":"127.0.0.1:0","remote_ip":"127.0.0.1","status":200,"system":"http","time":"2020-02-06T09:55:08+01:00","uri":"/group-name/project-name/-/environments/35/terminal.ws","user_agent":"Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/79.0.3945.130 Safari/537.36","written_bytes":35445} {"correlation_id":"tR8ggYTm3q4","encoding":"gzip","file":"/opt/gitlab/embedded/service/gitlab-rails/public/assets/webpack/pages.projects.environments.terminal.ff571abf.chunk.js.map","level":"info","method":"GET","msg":"Send static file","time":"2020-02-06T09:55:10+01:00","uri":"/assets/webpack/pages.projects.environments.terminal.ff571abf.chunk.js.map"} {"correlation_id":"tR8ggYTm3q4","duration_ms":4,"host":"gitlab.mysite.com","level":"info","method":"GET","msg":"access","proto":"HTTP/1.1","referrer":"","remote_addr":"127.0.0.1:0","remote_ip":"127.0.0.1","status":200,"system":"http","time":"2020-02-06T09:55:10+01:00","uri":"/assets/webpack/pages.projects.environments.terminal.ff571abf.chunk.js.map","user_agent":"Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/79.0.3945.130 Safari/537.36","written_bytes":271} {"correlation_id":"ZTlaJvCrrL9","encoding":"gzip","file":"/opt/gitlab/embedded/service/gitlab-rails/public/assets/webpack/vendors~select2.c6af6b5d.chunk.js.map","level":"info","method":"GET","msg":"Send static file","time":"2020-02-06T09:55:10+01:00","uri":"/assets/webpack/vendors~select2.c6af6b5d.chunk.js.map"} {"correlation_id":"ZTlaJvCrrL9","duration_ms":0,"host":"gitlab.mysite.com","level":"info","method":"GET","msg":"access","proto":"HTTP/1.1","referrer":"","remote_addr":"127.0.0.1:0","remote_ip":"127.0.0.1","status":200,"system":"http","time":"2020-02-06T09:55:10+01:00","uri":"/assets/webpack/vendors~select2.c6af6b5d.chunk.js.map","user_agent":"Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/79.0.3945.130 Safari/537.36","written_bytes":55003} {"correlation_id":"kbdMBqt7FT1","encoding":"gzip","file":"/opt/gitlab/embedded/service/gitlab-rails/public/assets/icons-e91700f3f1ecff110fc2c35aa62aec8f2aad69d1bfb35844186a11175a79e25f.svg","level":"info","method":"GET","msg":"Send static file","time":"2020-02-06T09:55:10+01:00","uri":"/assets/icons-e91700f3f1ecff110fc2c35aa62aec8f2aad69d1bfb35844186a11175a79e25f.svg"} {"correlation_id":"kbdMBqt7FT1","duration_ms":3,"host":"gitlab.mysite.com","level":"info","method":"GET","msg":"access","proto":"HTTP/1.1","referrer":"https://gitlab.mysite.com/group-name/project-name/-/environments/35/terminal","remote_addr":"127.0.0.1:0","remote_ip":"127.0.0.1","status":200,"system":"http","time":"2020-02-06T09:55:10+01:00","uri":"/assets/icons-e91700f3f1ecff110fc2c35aa62aec8f2aad69d1bfb35844186a11175a79e25f.svg","user_agent":"Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/79.0.3945.130 Safari/537.36","written_bytes":25939} {"correlation_id":"i03OQG69qg4","duration_ms":0,"host":"gitlab.mysite.com","level":"info","method":"POST","msg":"access","proto":"HTTP/1.1","referrer":"","remote_addr":"xx.xxx.xxx.xx:0","remote_ip":"xx.xxx.xxx.xx","status":204,"system":"http","time":"2020-02-06T09:55:11+01:00","uri":"/api/v4/jobs/request","user_agent":"gitlab-runner 12.6.0 (12-6-stable; go1.13.4; linux/amd64)","written_bytes":0} {"correlation_id":"NSIH1hCE4X4","duration_ms":0,"host":"gitlab.mysite.com","level":"info","method":"POST","msg":"access","proto":"HTTP/1.1","referrer":"","remote_addr":"xx.xxx.xxx.xx:0","remote_ip":"xx.xxx.xxx.xx","status":204,"system":"http","time":"2020-02-06T09:55:14+01:00","uri":"/api/v4/jobs/request","user_agent":"gitlab-runner 12.6.0 (12-6-stable; go1.13.4; linux/amd64)","written_bytes":0} ``` *gitlab-ctl status* ``` run: alertmanager: (pid 9167) 242912s; run: log: (pid 472) 1969871s run: gitaly: (pid 30963) 1196173s; run: log: (pid 478) 1969871s run: gitlab-exporter: (pid 9170) 242912s; run: log: (pid 468) 1969871s run: gitlab-workhorse: (pid 9157) 242913s; run: log: (pid 481) 1969871s run: grafana: (pid 9188) 242912s; run: log: (pid 467) 1969871s run: logrotate: (pid 5787) 1706s; run: log: (pid 476) 1969871s run: nginx: (pid 9212) 242911s; run: log: (pid 479) 1969871s run: node-exporter: (pid 9221) 242910s; run: log: (pid 477) 1969871s run: postgres-exporter: (pid 9227) 242910s; run: log: (pid 480) 1969871s run: postgresql: (pid 12494) 1723701s; run: log: (pid 466) 1969871s run: prometheus: (pid 9230) 242910s; run: log: (pid 475) 1969871s run: redis: (pid 12517) 1723701s; run: log: (pid 469) 1969871s run: redis-exporter: (pid 9244) 242910s; run: log: (pid 473) 1969871s run: registry: (pid 9253) 242909s; run: log: (pid 7728) 1968965s run: sidekiq: (pid 9272) 242909s; run: log: (pid 470) 1969871s run: unicorn: (pid 9278) 242908s; run: log: (pid 471) 1969871s ``` ### Output of checks This bug happens on a private gitlab-ce #### Results of GitLab environment info <details> <summary>Expand for output related to GitLab environment info</summary> <pre> System information System: Debian 9.11 Current User: git Using RVM: no Ruby Version: 2.6.5p114 Gem Version: 2.7.10 Bundler Version:1.17.3 Rake Version: 12.3.3 Redis Version: 5.0.7 Git Version: 2.24.1 Sidekiq Version:5.2.7 Go Version: unknown GitLab information Version: 12.7.5 Revision: b0fe37ef58e Directory: /opt/gitlab/embedded/service/gitlab-rails DB Adapter: PostgreSQL DB Version: 10.9 URL: http://gitlab.mysite.com HTTP Clone URL: http://gitlab.mysite.com/some-group/some-project.git SSH Clone URL: git@git.repo.mysite.com:some-group/some-project.git Using LDAP: no Using Omniauth: yes Omniauth Providers: google_oauth2 GitLab Shell Version: 11.0.0 Repository storage paths: - default: /var/opt/gitlab/git-data/repositories GitLab Shell path: /opt/gitlab/embedded/service/gitlab-shell Git: /opt/gitlab/embedded/bin/git </pre> </details> #### Results of GitLab application Check <details> <summary>Expand for output related to the GitLab application check</summary> <pre> Checking GitLab subtasks ... Checking GitLab Shell ... GitLab Shell: ... GitLab Shell version >= 11.0.0 ? ... OK (11.0.0) Running /opt/gitlab/embedded/service/gitlab-shell/bin/check Internal API available: OK Redis available via internal API: OK gitlab-shell self-check successful Checking GitLab Shell ... Finished Checking Gitaly ... Gitaly: ... default ... OK Checking Gitaly ... Finished Checking Sidekiq ... Sidekiq: ... Running? ... yes Number of Sidekiq processes ... 1 Checking Sidekiq ... Finished Checking Incoming Email ... Incoming Email: ... Reply by email is disabled in config/gitlab.yml Checking Incoming Email ... Finished Checking LDAP ... LDAP: ... LDAP is disabled in config/gitlab.yml Checking LDAP ... Finished Checking GitLab App ... Git configured correctly? ... yes Database config exists? ... yes All migrations up? ... yes Database contains orphaned GroupMembers? ... no GitLab config exists? ... yes GitLab config up to date? ... yes Log directory writable? ... yes Tmp directory writable? ... yes Uploads directory exists? ... yes Uploads directory has correct permissions? ... yes Uploads directory tmp has correct permissions? ... yes Init script exists? ... skipped (omnibus-gitlab has no init script) Init script up-to-date? ... skipped (omnibus-gitlab has no init script) Projects have namespace: ... 8/1 ... yes 10/2 ... yes 10/3 ... yes 12/5 ... yes 13/7 ... yes 16/8 ... yes 19/9 ... yes 23/13 ... yes 20/14 ... yes 10/16 ... yes 10/18 ... yes 10/19 ... yes 28/22 ... yes 31/23 ... yes 32/24 ... yes 31/25 ... yes 20/26 ... yes 8/27 ... yes 8/28 ... yes 8/29 ... yes 38/30 ... yes 38/32 ... yes 45/33 ... yes 8/34 ... yes 48/35 ... yes 49/36 ... yes 50/37 ... yes 53/38 ... yes 54/40 ... yes 20/41 ... yes 8/42 ... yes 58/44 ... yes 46/45 ... yes 60/47 ... yes 61/48 ... yes 53/49 ... yes 49/50 ... yes 49/51 ... yes 63/55 ... yes 64/56 ... yes 65/57 ... yes 64/59 ... yes 39/62 ... yes 69/63 ... yes 70/65 ... yes 70/66 ... yes 74/67 ... yes 35/68 ... yes 69/69 ... yes 86/70 ... yes 88/72 ... yes 43/73 ... yes 156/74 ... yes 93/77 ... yes 94/78 ... yes 88/79 ... yes 96/80 ... yes 16/81 ... yes 43/82 ... yes 8/83 ... yes 96/84 ... yes 99/85 ... yes 107/87 ... yes 107/89 ... yes 107/90 ... yes 91/91 ... yes 91/92 ... yes 111/93 ... yes 114/94 ... yes 114/95 ... yes 115/96 ... yes 117/97 ... yes 117/98 ... yes 118/99 ... yes 118/100 ... yes 121/101 ... yes 107/102 ... yes 126/105 ... yes 38/106 ... yes 38/107 ... yes 132/108 ... yes 132/109 ... yes 133/110 ... yes 136/111 ... yes 138/112 ... yes 138/113 ... yes 140/114 ... yes 141/115 ... yes 143/116 ... yes 143/117 ... yes 143/118 ... yes 114/119 ... yes 114/120 ... yes 144/121 ... yes 144/122 ... yes 143/123 ... yes 147/124 ... yes 43/125 ... yes 148/127 ... yes 150/128 ... yes 150/131 ... yes 151/132 ... yes 155/135 ... yes 155/136 ... yes 150/138 ... yes 155/140 ... yes 159/141 ... yes 162/142 ... yes 162/143 ... yes 167/147 ... yes 167/148 ... yes 168/149 ... yes 168/150 ... yes 169/151 ... yes 169/152 ... yes 172/153 ... yes 172/154 ... yes 172/155 ... yes 16/156 ... yes 16/157 ... yes 174/158 ... yes 176/159 ... yes 178/160 ... yes 181/161 ... yes Redis version >= 2.8.0? ... yes Ruby version >= 2.5.3 ? ... yes (2.6.5) Git version >= 2.22.0 ? ... yes (2.24.1) Git user has default SSH configuration? ... yes Active users: ... 52 Is authorized keys file accessible? ... yes Checking GitLab App ... Finished Checking GitLab subtasks ... Finished </pre> </details>