Skip to content

stdout and stderr streams not behaving correctly

MR:

Description

Alexandr Evstigneev and Nikolay Kuznetsov from JetBrains were helping us figure out why was JetBrains Gateway not working correctly when we try to use it for a workspace. In that process, we noticed that there is something off about gitlab-workspaces-proxy around how it is handling requests, stdout, stderr, etc.

Steps to reproduce

  1. Create a workspace on GitLab.com
  2. Run the following command on your local terminal
ssh WORKSPACE_NAME@LOAD_BALANCER_IP_OR_DOMAIN uname -sm
ssh WORKSPACE_NAME@LOAD_BALANCER_IP_OR_DOMAIN uname -sm 2> /dev/null
  1. When prompted for a password, enter a PAT with at least read_api access.

Notice the different output for the same commands and how stderr and stdout are not looking right.

➜  gitlab git:(master) ✗ ssh workspace-73241-10764887-chlhxq@199.223.234.75 uname -sm
^C
➜  gitlab git:(master) ✗ ssh workspace-73241-10764887-chlhxq@199.223.234.75 uname -sm
workspace-73241-10764887-chlhxq@199.223.234.75's password:
client_global_hostkeys_prove_confirm: server gave bad signature for RSA key 0: incorrect signature
➜  gitlab git:(master) ✗ ssh workspace-73241-10764887-chlhxq@199.223.234.75 uname -sm 2> /dev/null
workspace-73241-10764887-chlhxq@199.223.234.75's password:
Linux x86_64
➜  gitlab git:(master) ✗ ssh workspace-73241-10764887-chlhxq@199.223.234.75 uname -sm 1> /dev/null
workspace-73241-10764887-chlhxq@199.223.234.75's password:
client_global_hostkeys_prove_confirm: server gave bad signature for RSA key 0: incorrect signature
➜  gitlab git:(master) ✗ ssh workspace-73241-10764887-chlhxq@199.223.234.75 uname -sm 2> /dev/null
workspace-73241-10764887-chlhxq@199.223.234.75's password:
➜  gitlab git:(master) ✗ ssh workspace-73241-10764887-chlhxq@199.223.234.75 uname -sm 2> /dev/null
workspace-73241-10764887-chlhxq@199.223.234.75's password:
Linux x86_64
➜  gitlab git:(master) ✗ ssh workspace-73241-10764887-chlhxq@199.223.234.75 uname -sm 2> /dev/null
workspace-73241-10764887-chlhxq@199.223.234.75's password:
➜  gitlab git:(master) ✗

Notes from JetBrains folks

could you please try to run commands on your local machine: ssh @ uname -sm ssh @ uname -sm 2> /dev/null and check whether the output comes on stdout or stderr. where user and hosts are for the remote machine you are configuring

Gateway makes use of exec and signal requests. Make sure your proxy supports both, honors/proxies PTY request flag, properly forwards both stdout and stderr streams without truncating them, and provides process exit code. Quick inspection of the code doesn't show anything obviously wrong with proxying bits, but I'm not familiar with these specific go libraries and whether the cancellation logic on connection close there guarantees delivery of all requests and stream data.

Based on snippets with manual ssh command execution missing some replies, truncation of streams might be what's going on here.

Other weird things noticed

  • When running everything on local(Mac), things work fine.
  • When running Kubernetes cluster on a linux machine on a remote server, this issue starts popping up.
  • Maybe this is because this causes some delay in the network and this triggers some race condition or concurrency problem?
Edited by Vishal Tak