Skip to content
Snippets Groups Projects

Update Puma to 6.4.3 and patch PID reaper

Merged Stan Hu requested to merge sh-update-and-patch-puma into master

What does this MR do and why?

Puma v6.4.1 was reverted in #437948 (closed) since it exposed a PID-reaping bug in Ruby 3.1.4 (https://bugs.ruby-lang.org/issues/19837). Full details are in https://github.com/puma/puma/issues/3313.

Unfortunately Ruby 3.1.5 and 3.2.5 still have a bug that prevents Puma from reaping child PIDs properly (https://bugs.ruby-lang.org/issues/20490), so let's pull in the patch in https://github.com/puma/puma/pull/3314 since a Puma release has not been forthcoming.

This update fixes a minor security issue and other bugs: https://my.diffend.io/gems/puma/6.4.0/6.4.3

Related incidents:

MR acceptance checklist

Please evaluate this MR against the MR acceptance checklist. It helps you analyze changes to reduce risks in quality, performance, reliability, security, and maintainability.

How to set up and validate locally

GDK

  1. Check out this branch.
  2. gdk restart rails-web
  3. When Puma starts, kill one of the cluster worker jobs:
% ps -ef | grep "cluster worker"
  501 65022 64509   0 10:08PM ??         0:00.38 puma: cluster worker 0: 64509 [gitlab-puma-worker]
  501 65378 64509   0 10:08PM ??         0:00.40 puma: cluster worker 1: 64509 [gitlab-puma-worker]
  501 79824  7003   0 10:17PM ttys004    0:00.00 grep cluster worker
% kill -9 65378

You should see a new PID take its place immediately:

 % ps -ef | grep "cluster worker"
  501 65022 64509   0 10:08PM ??         0:00.39 puma: cluster worker 0: 64509 [gitlab-puma-worker]
  501 80175 64509   0 10:18PM ??         0:00.37 puma: cluster worker 1: 64509 [gitlab-puma-worker]
  501 80213  7003   0 10:18PM ttys004    0:00.00 grep cluster worker

With Cloud Native GitLab

  1. Build a webservice container with Puma 6.4.3. I used this Dockerfile:
FROM registry.gitlab.com/gitlab-org/build/cng/gitlab-webservice-ee:master

USER root
RUN apt update && apt install -y build-essential && cd /srv/gitlab && sed -i -e "s/= 6.4.0/= 6.4.3/" Gemfile && bundle config set frozen false && bundle install && bundle config set frozen true

COPY config/initializers/puma_patch.rb /srv/gitlab/config/initializers/puma_patch.rb

USER git
  1. Push this image to some public location. For example:
docker build -t registry.gitlab.com/stanhu/lfs-test/webservice-ee:latest .
docker push registry.gitlab.com/stanhu/lfs-test/webservice-ee:latest

If you are running this from your gitlab dir, be aware that .dockerignore may interfere with the above.

  1. Install the Helm Chart with this image:
helm upgrade --install gitlab . \
     --set gitlab.webservice.image.pullPolicy=Always \
     --set gitlab.webservice.image.repository=registry.gitlab.com/stanhu/lfs-test/webservice-ee \
     --set gitlab.webservice.image.tag=latest
  1. When the webservice pod comes up, log into it and verify that Puma 6.4.3 is running:
git@gitlab-webservice-default-66f688f648-g6prb:/$ ps -ef
UID          PID    PPID  C STIME TTY          TIME CMD
git            1       0 87 20:09 ?        00:00:39 puma 6.4.2 (tcp://0.0.0.0:8080) [gitlab-puma-worker]
git           21       1  0 20:09 ?        00:00:00 /usr/local/bin/gitlab-logger /var/log/gitlab
git           68       0  0 20:09 pts/0    00:00:00 bash
git           81       1 65 20:09 ?        00:00:02 ruby /srv/gitlab/bin/metrics-server
git           84       1 22 20:09 ?        00:00:00 puma: cluster worker 0: 1 [gitlab-puma-worker]
git           86       1 16 20:09 ?        00:00:00 puma: cluster worker 1: 1 [gitlab-puma-worker]
git          121      68  0 20:09 pts/0    00:00:00 ps -ef
  1. Then start killing the cluster worker PIDs and verify they are reaped. Without the patch, you will see a zombie process:
git@gitlab-webservice-default-66f688f648-g6prb:/$ kill -9 86
git@gitlab-webservice-default-66f688f648-g6prb:/$ ps -ef
UID          PID    PPID  C STIME TTY          TIME CMD
git            1       0 45 20:09 ?        00:00:39 puma 6.4.2 (tcp://0.0.0.0:8080) [gitlab-puma-worker]
git           21       1  0 20:09 ?        00:00:00 /usr/local/bin/gitlab-logger /var/log/gitlab
git           68       0  0 20:09 pts/0    00:00:00 bash
git           81       1  4 20:09 ?        00:00:02 ruby /srv/gitlab/bin/metrics-server
git           84       1  1 20:09 ?        00:00:00 puma: cluster worker 0: 1 [gitlab-puma-worker]
git           86       1  1 20:09 ?        00:00:00 [ruby] <defunct>
git          127      68  0 20:10 pts/0    00:00:00 ps -ef

With the patch, it works:

git@gitlab-webservice-default-6887959876-mg9mb:/srv/gitlab/config/initializers$ ps -ef | grep puma
git            1       0 88 05:12 ?        00:00:32 puma 6.4.3 (tcp://0.0.0.0:8080) [gitlab-puma-worker]
git           53       1 19 05:13 ?        00:00:00 puma: cluster worker 0: 1 [gitlab-puma-worker]
git           55       1 19 05:13 ?        00:00:00 puma: cluster worker 1: 1 [gitlab-puma-worker]
git           89      35  0 05:13 pts/0    00:00:00 grep puma
git@gitlab-webservice-default-6887959876-mg9mb:/srv/gitlab/config/initializers$ kill 55
git@gitlab-webservice-default-6887959876-mg9mb:/srv/gitlab/config/initializers$ ps -ef
UID          PID    PPID  C STIME TTY          TIME CMD
git            1       0 80 05:12 ?        00:00:32 puma 6.4.3 (tcp://0.0.0.0:8080) [gitlab-puma-worker]
git           21       1  0 05:12 ?        00:00:00 /usr/local/bin/gitlab-logger /var/log/gitlab
git           35       0  0 05:13 pts/0    00:00:00 bash
git           50       1 29 05:13 ?        00:00:01 ruby /srv/gitlab/bin/metrics-server
git           53       1  7 05:13 ?        00:00:00 puma: cluster worker 0: 1 [gitlab-puma-worker]
git           92       1 29 05:13 ?        00:00:00 puma: cluster worker 1: 1 [gitlab-puma-worker]
git          109      35  0 05:13 pts/0    00:00:00 ps -ef
Edited by Stan Hu

Merge request reports

Merge train pipeline #1467433473 passed

Merge train pipeline passed for edd7f134

Merged by Stan HuStan Hu 6 months ago (Sep 24, 2024 5:02pm UTC)

Loading

Pipeline #1467435986 passed with warnings

Pipeline passed with warnings for 73b827ac on master

10 environments impacted.

Activity

Filter activity
  • Approvals
  • Assignees & reviewers
  • Comments (from bots)
  • Comments (from users)
  • Commits & branches
  • Edits
  • Labels
  • Lock status
  • Mentions
  • Merge request status
  • Tracking
Please register or sign in to reply
Loading