Skip Topology Service claim warmup outside Puma and Sidekiq

Reuben Pereirarequested to merge
rp-skip-claim-warmup-outside-servers into master

What does this MR do and why?

On Cells, config/initializers/topology_service_claim_warmup.rb opens a gRPC connection to the Topology Service using on_worker_start. In clustered Puma that block runs in each worker after it forks, which is safe. In every other process it runs straight away during boot.

This breaks processes like gitlab-rails runner that boot once and then fork. Each forked child gets a copy of the gRPC objects but not the gRPC threads they rely on. When the child exits, Ruby frees those objects and gRPC's native code crashes. The child then hangs until it's killed.

Instrumentor's driver.rb works this way: it boots Rails once and forks a child per support script. So the configure script in Instrumentor fails on every Cell run with current instrumentor versions. Dedicated isn't affected because it doesn't set cell.enabled.

This MR only runs the warmup in Puma and Sidekiq (using Gitlab::Runtime.application?). The warmup only speeds up the first request in processes that serve requests, so skipping it everywhere else should be safe.

Related to https://gitlab.com/gitlab-com/gl-infra/gitlab-dedicated/team/-/work_items/14350

References

Screenshots or screen recordings

Before After

How to set up and validate locally

Reproducing the crash needs a Cells environment with a reachable Topology Service. I reproduced it in a cellsdev toolbox pod:

  1. Run a gitlab-rails runner test script that forks 5 children. Each child runs SELECT 1 and exits normally.
  2. With the warmup running, 4 of 5 children crashed.
  3. With ClaimService#warmup! replaced by a no-op, 0 of 5 crashed.

MR acceptance checklist

Evaluate this MR against the MR acceptance checklist. It helps you analyze changes to reduce risks in quality, performance, reliability, security, and maintainability.

Edited by Reuben Pereira

Merge request reports