Interactive Web terminal self-hosted

Summary

I'm currently trying to setup the nice Web-Terminal feature on my self-hosted Gitlab-Instance. But I can't get it running properly. Also debugging is a rather difficult task.

Environment description/Steps to reproduce

Gitlab version: GitLab Community Edition 13.9.0 Gitlab runner on the same machine as Gitlab: Runs in a docker-container with multiple runners configured. Output of docker ps (port 8093 is forwarded correctly):

CONTAINER ID   IMAGE                         COMMAND                  CREATED        STATUS        PORTS                                                                                                                                                                                                                               NAMES
CONTAINERID    gitlab/gitlab-runner:latest   "/usr/bin/dumb-init …"   15 hours ago   Up 12 hours   0.0.0.0:8093->8093/tcp                                                                                                                                                                                                              gitlab-runner2

The config.toml is placed in /srv/gitlab-runner/config/config.toml (from the gitlab-docs and I added the following configuration (from the gitlab-docs):

[session_server]
  listen_address = "[::]:8093"
  advertise_address = "localhost:8093"
  session_timeout = 1800

The official docs state that the advertise_address is not needed necessarily and it falls back to the listen_address. Nevertheless I read in several issues here on that topic, that this needs to be the external IP/hostname, where the runners are accessible. In my case they do not need any external IP as they are running on the same machine. Nevertheless I also tried it with my static server-ip and the domain of my selfhosted gitlab without success.

Gitlab is running behind an Apache-Proxy. The Gitlab-docs do not offer a lot information about how the proxy should be configured for the interactive terminals.

My current Apache-config for gitlab looks like this (gitlab is running on port 7777):

Apache configuration

``` ServerName git.example.com

ProxyPreserveHost On
ProxyRequests     Off
RewriteEngine on
SSLProxyEngine On
AllowEncodedSlashes NoDecode

<Location />
    ProxyPass http://localhost:7777/ nocanon
    ProxyPassReverse https://git.example.com/
    ProxyPassReverse http://localhost:7777/
    Require all granted
</Location>

RewriteCond %{HTTP:Upgrade} websocket [NC]
RewriteCond %{HTTP:Connection} upgrade [NC]
RewriteRule .* "wss:/localhost:7777/$1" [P,L]

ProxyPass "*/terminal.ws" "wss://localhost:7777/"
<Location */terminal.ws>
    RewriteCond %{HTTP:Upgrade} websocket [NC]
    RewriteCond %{HTTP:Connection} upgrade [NC]
    RewriteRule ^/?(.*) "wss://localhost:7777/$1" [P,L]
 </Location>

<VirtualHost *:443> ServerName pages.example.com ServerAlias *.pages.example.com

ProxyPreserveHost On
ProxyPass "/" "http://127.0.0.1:8090/"
ProxyPassReverse "/" "http://127.0.0.1:8090/

RewriteEngine on
RewriteCond %{DOCUMENT_ROOT}/%{REQUEST_FILENAME} !-f
RewriteRule .* http://127.0.0.1:8090%{REQUEST_URI} [P,QSA]

```

Here are my relevant changes in the gitlab.rb-file:

Gitlab configuration

nginx['listen_addresses'] = ['0.0.0.0', "[::]"]
nginx['listen_port'] = 7777
nginx['listen_https'] = false
puma['port'] = 8081
external_url 'https://git.example.com'
pages_external_url "http://pages.example.com/"
gitlab_pages['enable'] = true
gitlab_pages['listen_proxy'] = "localhost:8090"

gitlab_pages['redirect_http'] = false
pages_nginx['enable'] = false

I also tried adding this to the gitlab.rb but I think this is the default configuration anyway:

nginx['proxy_set_headers'] = {
  "Host" => "$http_host_with_default",
  "X-Real-IP" => "$remote_addr",
  "X-Forwarded-For" => "$proxy_add_x_forwarded_for",
  "X-Forwarded-Proto" => "https",
  "X-Forwarded-Ssl" => "on",
  "Upgrade" => "$http_upgrade",
  "Connection" => "$connection_upgrade"
}

This is my .gitlab/.gitlab-webide.yml of the project, which I want to use:

terminal:
  image:
    name: python:latest
  script: sleep 60

I also tried to disable my firewall (ufw) and allowed connections on port 8093 without success.

Actual behavior

Gitlab pages are working fine, also the configured runners are working fine in pipelines. But when I click either the "Debug" option for running jobs or the Web-Terminal, a connection error occurs:

terminal.js:47 WebSocket connection to 'wss://git.example.com/user/project/-/jobs/JOB_ID/terminal.ws' failed: Error during WebSocket handshake: Unexpected response code: 404

But the Job for the Webterminal is actually running (the sleep 60 line runs, after this the pipeline succeeded). I can view the log of the job at https://git.example.com/user/project/-/jobs/JOB_ID.

After searching a lot, I now don't know how to proceed - especially because I have no good debug-strategy.

Edit

I tried to connect with the server with websocat wss://0.0.0.0:8094 which returns

websocat: WebSocket SSL error: error:1416F086:SSL routines:tls_process_server_certificate:certificate verify failed:../ssl/statem/statem_clnt.c:1915: (self signed certificate)
websocat: error running

When I ignore the self signed cert with websocat -t - ws-c:sh-c:'socat - ssl:0.0.0.0:8093,verify=0' --ws-c-uri=wss://echo.websocket.org (some workaround for websocat) it returns the same error as in the WebIDE:

websocat: WebSocketError: Received unexpected status code (404 Not Found)
websocat: error running

Expected behavior

The terminal should be visible in the browser or any outputs in any log-file.

Used GitLab Runner version

Version:      13.8.0
Git revision: 775dd39d
Git branch:   13-8-stable
GO version:   go1.13.8
Built:        2021-01-20T13:32:47+0000
OS/Arch:      linux/amd64