Trap kill signal and call cleanup/stop

What does this MR do and why?

In the context of GDK, when running gdk stop or gdk stop rails-background-jobs, the TERM signal is sent to the bin/background_jobs script. Without this change, bin/background_jobs is immediately killed but the sidekiq / sidekiq-cluster processes remain, as orphans/zombies.

This MR traps when the TERM signal is sent to bin/background_jobs and runs the cleanup() function before exiting. cleanup() currently calls the stop() function to ensure a proper shutdown for sidekiq / sidekiq-cluster.

Closes: gitlab-development-kit#1344 (closed)

How to set up and validate locally

1. Ensure you're sitting in your GDK root directory and you're up-to-date:

   cd <your-GDK-root> && git fetch && git checkout main

2. Ensure a clean slate:

   gdk stop && gdk kill && pkill -9 sidekiq

   You may need to run a few times to ensure there's no sidekiq or sidekiq-cluster processes.

3. Ensure you don't see sidekiq running:

   ps -ef | grep sidekiq

4. Ensure you have the bare minimum 'DB' services and rails-background-jobs running:

   gdk start db rails-background-jobs

5. Ensure you can see sidekiq running:

   ps -ef | grep sidekiq

6. Take note of the output from and save it somewhere for reference.

7. In your GDK_ROOT/gitlab/ directory, check out the ashmckenzie/trap-kill-and-cleanup-for-sidekiq branch:

   cd <your-GDK-root>/gitlab/ && git fetch && git checkout ashmckenzie/trap-kill-and-cleanup-for-sidekiq

8. Stop rails-background-jobs:

   gdk stop rails-background-jobs

9. Ensure you don't see sidekiq running (give it a few seconds):

   ps -ef | grep sidekiq

Contrast the above steps using origin/master in your GDK_ROOT/gitlab/ and you'll see 'orphan' sidekiq / sidekiq-cluster processes. An orphan/zombie process is one that has 'lost' its parent process and is identifiable by the parent process ID (PPID) being 1.

MR acceptance checklist

This checklist encourages us to confirm any changes have been analyzed to reduce risks in quality, performance, reliability, security, and maintainability.

• I have evaluated the MR acceptance checklist for this MR.

changed milestone to %14.5

added Category:GDK Engineering Productivity Quality + 1 deleted label

assigned to @ashmckenzie

added 1 deleted label

added typefeature label

marked the checklist item I have evaluated the MR acceptance checklist for this MR. as completed

added backend label

	1 Message
	CHANGELOG missing: If you want to create a changelog entry for GitLab FOSS, add the Changelog trailer to the commit message you want to add to the changelog. If you want to create a changelog entry for GitLab EE, also add the EE: true trailer to your commit message. If this merge request doesn't need a CHANGELOG entry, feel free to ignore this message.

Reviewer roulette

Changes that require review have been detected!

Please refer to the table below for assigning reviewers and maintainers suggested by Danger in the specified category:

Category	Reviewer	Maintainer
backend	Niko Belokolodov (@nbelokolodov) (UTC+13, 2 hours ahead of @ashmckenzie)	Terri Chu (@terrichu) (UTC-4, 15 hours behind @ashmckenzie)

To spread load more evenly across eligible reviewers, Danger has picked a candidate for each review slot, based on their timezone. Feel free to override these selections if you think someone else would be better-suited or use the GitLab Review Workload Dashboard to find other available reviewers.

To read more on how to use the reviewer roulette, please take a look at the Engineering workflow and code review guidelines. Please consider assigning a reviewer or maintainer who is a domain expert in the area of the merge request.

Once you've decided who will review this merge request, assign them as a reviewer! Danger does not automatically notify them for you.

If needed, you can retry the danger-review job that generated this comment.

Generated by Danger

changed the description

removed backend label

mentioned in issue gitlab-development-kit#1344 (closed)

@bdenkovych as mentioned over at gitlab-development-kit#1344 (comment 719605146), may you please follow the steps to validate locally and also review this MR please?

If it works and you're happy, may you please assign to @terrichu for maintainer review please?

requested review from @bdenkovych

mentioned in merge request gitlab-development-kit!2236 (closed)

changed the description

@terrichu Could you please review this MR as a maintainer?

requested review from @terrichu

approved this merge request

@bdenkovych, thanks for approving this merge request.

This is the first time the merge request is approved. To ensure full test coverage, a new pipeline has been started.

For more info, please refer to the following links:

• rspec minimal jobs
• jest minimal jobs
• merging a merge request.

added backend label

@ashmckenzie I tried the testing steps and am still getting what appears to be an orphaned sidekiq-cluster process. I'm on a mac and can provide any other logs you need, let me know!

before stopping rails-background-jobs

~/Projects/gitlab/gdk on  main [$] via  v14.17.5 via 💎 v2.7.4 via ⍱ at 
➜ ps -ef | grep sidekiq
  501 59637 59551   0 10:44AM ??         0:00.18 ruby bin/sidekiq-cluster default,mailers,email_receiver,hashed_storage:hashed_storage_migrator,hashed_storage:hashed_storage_project_migrate,hashed_storage:hashed_storage_project_rollback,hashed_storage:hashed_storage_rollbacker,project_import_schedule,service_desk_email_receiver -P /Users/terrichu/Projects/gitlab/gdk/gitlab/tmp/pids/sidekiq-cluster.pid -e development --timeout 10
  501 59639 59551   0 10:44AM ??         0:00.00 tee -a /Users/terrichu/Projects/gitlab/gdk/gitlab/log/sidekiq.log
  501 59851 59637   0 10:44AM ??         0:09.42 /Users/terrichu/.asdf/installs/ruby/2.7.4/bin/sidekiq -c10 -edevelopment -t10 -gqueues:default,mailers,email_receiver,hashed_storage:hashed_storage_migrator,hashed_storage:hashed_storage_project_migrate,hashed_storage:hashed_storage_project_rollback,hashed_storage:hashed_storage_rollbacker,project_import_schedule,service_desk_email_receiver -r/Users/terrichu/Projects/gitlab/gdk/gitlab -qdefault,1 -qmailers,1 -qemail_receiver,1 -qhashed_storage:hashed_storage_migrator,1 -qhashed_storage:hashed_storage_project_migrate,1 -qhashed_storage:hashed_storage_project_rollback,1 -qhashed_storage:hashed_storage_rollbacker,1 -qproject_import_schedule,1 -qservice_desk_email_receiver,1
  501 60804   784   0 10:44AM ttys002    0:00.00 grep --color=auto --exclude-dir=.bzr --exclude-dir=CVS --exclude-dir=.git --exclude-dir=.hg --exclude-dir=.svn --exclude-dir=.idea --exclude-dir=.tox sidekiq

after stopping rails-background-jobs

~/Projects/gitlab/gdk/gitlab on  ashmckenzie/trap-kill-and-cleanup-for-sidekiq [$] via  v14.17.5 via 🐍 v2.7.16 via 💎 v2.7.4 at 
➜ ps -ef | grep sidekiq
  501 59637     1   0 10:44AM ??         0:00.19 ruby bin/sidekiq-cluster default,mailers,email_receiver,hashed_storage:hashed_storage_migrator,hashed_storage:hashed_storage_project_migrate,hashed_storage:hashed_storage_project_rollback,hashed_storage:hashed_storage_rollbacker,project_import_schedule,service_desk_email_receiver -P /Users/terrichu/Projects/gitlab/gdk/gitlab/tmp/pids/sidekiq-cluster.pid -e development --timeout 10
  501 59851 59637   0 10:44AM ??         0:20.27 sidekiq 6.2.2 queues:default,mailers,email_receiver,hashed_storage:hashed_storage_migrator,hashed_storage:hashed_storage_project_migrate,hashed_storage:hashed_storage_project_rollback,hashed_storage:hashed_storage_rollbacker,project_import_schedule,service_desk_email_receiver [0 of 10 busy]
  501 65619   872   0 10:45AM ttys003    0:00.00 grep --color=auto --exclude-dir=.bzr --exclude-dir=CVS --exclude-dir=.git --exclude-dir=.hg --exclude-dir=.svn --exclude-dir=.idea --exclude-dir=.tox sidekiq

removed review request for @terrichu

approved this merge request

requested review from @terrichu

resolved all threads

enabled an automatic merge when the pipeline for 8ccd2e23 succeeds

resolved all threads

merged

mentioned in commit 1d75928a

added workflowstaging-canary label

added workflowstaging label and removed workflowstaging-canary label

added workflowcanary label and removed workflowstaging label

mentioned in issue gitlab-development-kit#1348 (closed)

added workflowproduction label and removed workflowcanary label

added typemaintenance label and removed typefeature + 1 deleted label

added typebug label and removed typemaintenance + 1 deleted label

added releasedcandidate label

added releasedpublished label and removed releasedcandidate label

mentioned in issue gitlab-development-kit#1376 (closed)

mentioned in merge request gitlab-com/www-gitlab-com!112458 (merged)