`gdk stop rails-background-jobs` makes orphan sidekiq process

Overview

After gdk stop rails-background-jobs, the orphan sidekiq process is created and it consumes almost all RAM of developers laptop.

See https://app.slack.com/client/T02592416/C2Z9A056E/thread/C2Z9A056E-1634733859.058600 and https://app.slack.com/client/T02592416/C2Z9A056E.

If developers want to restart or stop the sidekiq, they need to reboot their laptop to prevent this issue. It makes the dev experience quite inconvenient.

Steps to replicate (optional)

  1. gdk start rails-background-jobs

  2. ps axjf | grep -E -A 5 -B 5 '(rails-background-jobs|background_jobs|sidekiq)' outputs:

    PPID     PID    PGID     SID TTY        TPGID STAT   UID   TIME COMMAND
 229826  229835  229835  229835 ?             -1 Ss    1000   0:00  |   \_ runsv rails-background-jobs
 229835  229846  229835  229835 ?             -1 S     1000   0:00  |   |   \_ svlogd -tt /home/bogdanvlviv/gitlab-development-kit/log/rails-background-jobs
 229835  292492  292492  292492 ?             -1 Ss    1000   0:00  |   |   \_ bash bin/background_jobs start_foreground --timeout 10
 292492  292499  292499  292499 ?             -1 Ssl   1000   0:00  |   |       \_ ruby bin/sidekiq-cluster default,mailers,email_receiver,hashed_storage:hashed_storage_migrator,hashed_storage:hashed_storage_project_migrate,hashed_storage:hashed_storage_project_rollback,hashed_storage:hashed_storage_rollbacker,project_import_schedule,service_desk_email_receiver -P /home/bogdanvlviv/gitlab-development-kit/gitlab/tmp/pids/sidekiq-cluster.pid -e development --timeout 10
 292499  292501  292501  292499 ?             -1 Sl    1000   0:09  |   |       |   \_ sidekiq 6.2.2 queues:default,mailers,email_receiver,hashed_storage:hashed_storage_migrator,hashed_storage:hashed_storage_project_migrate,hashed_storage:hashed_storage_project_rollback,hashed_storage:hashed_storage_rollbacker,project_import_schedule,service_desk_email_receiver [0 of 10 busy]
 292492  292500  292492  292492 ?             -1 S     1000   0:00  |   |       \_ tee -a /home/bogdanvlviv/gitlab-development-kit/gitlab/log/sidekiq.log

  1. gdk stop rails-background-jobs

  2. ps axjf | grep -E -A 5 -B 5 '(rails-background-jobs|background_jobs|sidekiq)' outputs orphan sidekiq processes:

   PPID     PID    PGID     SID TTY        TPGID STAT   UID   TIME COMMAND
 229826  229835  229835  229835 ?             -1 Ss    1000   0:00  |   \_ runsv rails-background-jobs
 229835  229846  229835  229835 ?             -1 S     1000   0:00  |   |   \_ svlogd -tt /home/bogdanvlviv/gitlab-development-kit/log/rails-background-jobs
 229826  229836  229836  229836 ?             -1 Ss    1000   0:00  |   \_ runsv postgresql
 229836  229845  229836  229836 ?             -1 S     1000   0:00  |       \_ svlogd -tt /home/bogdanvlviv/gitlab-development-kit/log/postgresql
 229836  240723  240723  240723 ?             -1 Ss    1000   0:00  |       \_ ruby support/postgresql-signal-wrapper /home/bogdanvlviv/.asdf/shims/postgres -D /home/bogdanvlviv/gitlab-development-kit/postgresql/data -k /home/bogdanvlviv/gitlab-development-kit/postgresql -h
 240723  240837  240723  240723 ?             -1 S     1000   0:01  |           \_ /home/bogdanvlviv/.asdf/installs/postgres/12.6/bin/postgres -D /home/bogdanvlviv/gitlab-development-kit/postgresql/data -k /home/bogdanvlviv/gitlab-development-kit/postgresql -h
 240837  241428  241428  241428 ?             -1 Ss    1000   0:00  |               \_ postgres: checkpointer
--
 240837  292590  292590  292590 ?             -1 Ss    1000   0:00  |               \_ postgres: bogdanvlviv gitlabhq_development [local] idle
 240837  292681  292681  292681 ?             -1 Ss    1000   0:00  |               \_ postgres: bogdanvlviv gitlabhq_development [local] idle
   1909  278192    1924    1924 ?             -1 Sl    1000   0:00  \_ /usr/bin/gnome-calendar --gapplication-service
   1909  279015  279015  279015 ?             -1 Ssl   1000   0:00  \_ /usr/libexec/flatpak-session-helper
   1909  279019  279019  279019 ?             -1 Ss    1000   0:00  \_ server --sh -n /run/user/1000/.flatpak-helper/pkcs11-flatpak-279015 --provider p11-kit-trust.so pkcs11:model=p11-kit-trust?write-protected=yes
   1909  292499  292499  292499 ?             -1 Ssl   1000   0:00  \_ ruby bin/sidekiq-cluster default,mailers,email_receiver,hashed_storage:hashed_storage_migrator,hashed_storage:hashed_storage_project_migrate,hashed_storage:hashed_storage_project_rollback,hashed_storage:hashed_storage_rollbacker,project_import_schedule,service_desk_email_receiver -P /home/bogdanvlviv/gitlab-development-kit/gitlab/tmp/pids/sidekiq-cluster.pid -e development --timeout 10
 292499  292501  292501  292499 ?             -1 Sl    1000   0:14      \_ sidekiq 6.2.2 queues:default,mailers,email_receiver,hashed_storage:hashed_storage_migrator,hashed_storage:hashed_storage_project_migrate,hashed_storage:hashed_storage_project_rollback,hashed_storage:hashed_storage_rollbacker,project_import_schedule,service_desk_email_receiver [0 of 10 busy]
  1. If repeat 1-4, we will see additiol orphan sidekiq process
   PPID     PID    PGID     SID TTY        TPGID STAT   UID   TIME COMMAND
 229826  229835  229835  229835 ?             -1 Ss    1000   0:00  |   \_ runsv rails-background-jobs
 229835  229846  229835  229835 ?             -1 S     1000   0:00  |   |   \_ svlogd -tt /home/bogdanvlviv/gitlab-development-kit/log/rails-background-jobs
 229826  229836  229836  229836 ?             -1 Ss    1000   0:00  |   \_ runsv postgresql
 229836  229845  229836  229836 ?             -1 S     1000   0:00  |       \_ svlogd -tt /home/bogdanvlviv/gitlab-development-kit/log/postgresql
 229836  240723  240723  240723 ?             -1 Ss    1000   0:00  |       \_ ruby support/postgresql-signal-wrapper /home/bogdanvlviv/.asdf/shims/postgres -D /home/bogdanvlviv/gitlab-development-kit/postgresql/data -k /home/bogdanvlviv/gitlab-development-kit/postgresql -h
 240723  240837  240723  240723 ?             -1 S     1000   0:01  |           \_ /home/bogdanvlviv/.asdf/installs/postgres/12.6/bin/postgres -D /home/bogdanvlviv/gitlab-development-kit/postgresql/data -k /home/bogdanvlviv/gitlab-development-kit/postgresql -h
 240837  241428  241428  241428 ?             -1 Ss    1000   0:00  |               \_ postgres: checkpointer
--
 240837  293938  293938  293938 ?             -1 Ss    1000   0:00  |               \_ postgres: bogdanvlviv gitlabhq_development [local] idle
 240837  293942  293942  293942 ?             -1 Ss    1000   0:00  |               \_ postgres: bogdanvlviv gitlabhq_development [local] idle
   1909  278192    1924    1924 ?             -1 Sl    1000   0:00  \_ /usr/bin/gnome-calendar --gapplication-service
   1909  279015  279015  279015 ?             -1 Ssl   1000   0:00  \_ /usr/libexec/flatpak-session-helper
   1909  279019  279019  279019 ?             -1 Ss    1000   0:00  \_ server --sh -n /run/user/1000/.flatpak-helper/pkcs11-flatpak-279015 --provider p11-kit-trust.so pkcs11:model=p11-kit-trust?write-protected=yes
   1909  292499  292499  292499 ?             -1 Ssl   1000   0:00  \_ ruby bin/sidekiq-cluster default,mailers,email_receiver,hashed_storage:hashed_storage_migrator,hashed_storage:hashed_storage_project_migrate,hashed_storage:hashed_storage_project_rollback,hashed_storage:hashed_storage_rollbacker,project_import_schedule,service_desk_email_receiver -P /home/bogdanvlviv/gitlab-development-kit/gitlab/tmp/pids/sidekiq-cluster.pid -e development --timeout 10
 292499  292501  292501  292499 ?             -1 Sl    1000   0:22  |   \_ sidekiq 6.2.2 queues:default,mailers,email_receiver,hashed_storage:hashed_storage_migrator,hashed_storage:hashed_storage_project_migrate,hashed_storage:hashed_storage_project_rollback,hashed_storage:hashed_storage_rollbacker,project_import_schedule,service_desk_email_receiver [0 of 10 busy]
   1909  293611  293611  293611 ?             -1 Ssl   1000   0:00  \_ ruby bin/sidekiq-cluster default,mailers,email_receiver,hashed_storage:hashed_storage_migrator,hashed_storage:hashed_storage_project_migrate,hashed_storage:hashed_storage_project_rollback,hashed_storage:hashed_storage_rollbacker,project_import_schedule,service_desk_email_receiver -P /home/bogdanvlviv/gitlab-development-kit/gitlab/tmp/pids/sidekiq-cluster.pid -e development --timeout 10
 293611  293613  293613  293611 ?             -1 Sl    1000   0:06      \_ sidekiq 6.2.2 queues:default,mailers,email_receiver,hashed_storage:hashed_storage_migrator,hashed_storage:hashed_storage_project_migrate,hashed_storage:hashed_storage_project_rollback,hashed_storage:hashed_storage_rollbacker,project_import_schedule,service_desk_email_receiver [0 of 10 busy]

Proposal (optional)

We should, somehow, prevent making this orphan sidekiq process on gdk stop.

https://en.wikipedia.org/wiki/Orphan_process

Environment (optional)

On Slack, people mention that it happens on both - Linux and macOS.

  • Operating system: Linux lenovo 5.10.0-1050-oem #52-Ubuntu SMP Wed Oct 6 12:58:49 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux
  • The contents of your gdk.yml 
hostname: gdk.test
port: 3443
https:
  enabled: true
nginx:
  enabled: true
  ssl:
    certificate: gdk.test.pem
    key: gdk.test-key.pem
  • Ruby version: ruby 2.7.4p191 (2021-07-07 revision a21a3b7d23) [x86_64-linux]
  • GDK version: 69f5ceea
Edited by Ash McKenzie