terminationGracePeriodSeconds not configurable for gitlab-shell
Summary
The gitlab-shell
pod does not cleanly exit. Upon asking for the stopping of this Pod, Kubernetes successfully removes the Pod from the Endpoints of the Service, but other processes remain, those include our process-wrapper script, sshd, and any ssh clients connected to the Pod. Because of the process-wrapper remaining, we currently rely on the terminationGracePeriod
to remove a Pod from the infrastructure.
Because terminationGracePeriod
value is not configurable, during deploys of the gitlab-shell
, we may unnecessarily cut active sessions (sessions that take quite a long period of time). I'd like to request that we make this value configurable in order to extend the time for long running connections to complete prior to Kubernetes taking the Pod out prematurely. This will still not cover clients that are operating for longer periods of time, but minimally allows us to provide a bit of configuration for the only current item that allows us to customize how pod deaths are handled.
Ideally we'd make this smarter such that the process wrapper is aware of the active connections and can complete the removal of the Pod prior to us hitting the terminiationGracePeriod
. Currently I'm unable to find any way to utilize sshd for this, so we may need to build better logic inside of the process wrapper script if we choose to go down a cleaner route.
Currently we have no visibility into how long git sessions through the sshd or gitlab-shell last.
Steps to reproduce
- With a working gitlab installation
- Push or pull a large repository, one that would take a few minutes
- During this period, initiate a (un)forced deletion of the gitlab-shell Pod
- The git operation on the client will have failed
Current behavior
Clients that are active are unable to complete their job.
Expected behavior
The Pod should stay around until clients have completed, or hitting the terminationGracePeriod
.
Relevant logs
Client side:
10s www-gitlab-com master % git push minikube master
Warning: Permanently added '[gitlab.172.17.0.3.nip.io]:32022' (ECDSA) to the list of known hosts.
Enumerating objects: 1134129, done.
Counting objects: 100% (1134129/1134129), done.
Delta compression using up to 8 threads
Compressing objects: 100% (347166/347166), done.
client_loop: send disconnect: Broken pipe61.01 MiB | 38.79 MiB/s
fatal: the remote end hung up unexpectedly
fatal: the remote end hung up unexpectedly