Sidekiq heartbeat starvation under load after Ruby 3.2.x → 3.3.10 upgrade (GitLab 18.8 → 18.9)

Everyone can contribute. Help move this issue forward while earning points, leveling up and collecting rewards.

Summary

After upgrading from GitLab 18.8 (Ruby 3.2.8) to 18.9 (Ruby 3.3.10), Sidekiq worker processes silently deregister from Redis under load. Processes remain alive at the OS level but show 0 on the Admin → Monitoring → Background Jobs → Busy tab. Jobs queue up and all Sidekiq-dependent functionality stops until a restart.

This issue has been confirmed on two separate self-managed instances (Podman/Quadlet deployment) with identical configurations but different hardware. No changes in user load or configuration were made between versions.

Steps to reproduce

  1. Upgrade GitLab from 18.8 (Ruby 3.2.x) to 18.9 (Ruby 3.3.10)
  2. Configure Sidekiq with moderate-to-high concurrency (e.g., sidekiq['queue_groups'] = ['*'] * 2 with 20 threads each per the default concurrency settings)
  3. Run under normal daytime load
  4. Monitor Admin → Monitoring → Background Jobs → Busy tab
  5. Observe Sidekiq processes showing 0 busy workers despite jobs queuing up

Example Project

N/A

What is the current bug behavior?

Sidekiq worker processes silently deregister from Redis under load. The processes remain alive at the OS level but are no longer visible in the GitLab admin interface. Jobs queue up and all Sidekiq-dependent functionality stops (CI pipelines, merge request updates, emails, webhooks, etc.) until a manual restart is performed:

gitlab-ctl restart sidekiq
timeout: run: sidekiq: (pid 403767) 150897s, got TERM

What is the expected correct behavior?

Sidekiq processes should maintain their Redis heartbeat and continue processing jobs under normal load without requiring manual intervention.

Relevant logs and/or screenshots

Thread dumps during failure state:

  • ThreadError: can't create Thread: Resource temporarily unavailable
/opt/gitlab/embedded/service/gitlab-rails/vendor/gems/sidekiq/lib/sidekiq/component.rb:17:in `initialize': can't create Thread: Resource temporarily unavailable (ThreadError)
        from /opt/gitlab/embedded/service/gitlab-rails/vendor/gems/sidekiq/lib/sidekiq/component.rb:17:in `new'
        from /opt/gitlab/embedded/service/gitlab-rails/vendor/gems/sidekiq/lib/sidekiq/component.rb:17:in `safe_thread'
        from /opt/gitlab/embedded/service/gitlab-rails/vendor/gems/sidekiq/lib/sidekiq/processor.rb:66:in `start'
        from /opt/gitlab/embedded/service/gitlab-rails/vendor/gems/sidekiq/lib/sidekiq/manager.rb:76:in `block in processor_result'
        from /opt/gitlab/embedded/service/gitlab-rails/vendor/gems/sidekiq/lib/sidekiq/manager.rb:71:in `synchronize'
        from /opt/gitlab/embedded/service/gitlab-rails/vendor/gems/sidekiq/lib/sidekiq/manager.rb:71:in `processor_result'
        from /opt/gitlab/embedded/service/gitlab-rails/vendor/gems/sidekiq/lib/sidekiq/processor.rb:81:in `rescue in run'
        from /opt/gitlab/embedded/service/gitlab-rails/vendor/gems/sidekiq/lib/sidekiq/processor.rb:71:in `run'
        from /opt/gitlab/embedded/service/gitlab-rails/vendor/gems/sidekiq/lib/sidekiq/component.rb:10:in `watchdog'
        from /opt/gitlab/embedded/service/gitlab-rails/vendor/gems/sidekiq/lib/sidekiq/component.rb:19:in `block in safe_thread'
  • Gitaly fork/exec: resource temporarily unavailable
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/git/wraps_gitaly_errors.rb:52:in `handle_default_error': 13:get default branch: starting process [/var/opt/gitlab/gitaly/run/gitaly-565/git-exec-484403296.d/git --git-dir /var/opt/gitlab/git-data/repositories/@hashed/36/58/3658d7fa3c43456f3c9c87db0490e872039516e6375336254560167cc3db2ea2.git -c gc.auto=0 -c maintenance.auto=0 -c core.autocrlf=false -c core.useReplaceRefs=false -c core.bigFileThreshold=50m -c pack.allowPackReuse=multi -c core.fsync=objects,derived-metadata,reference -c core.fsyncMethod=fsync -c core.packedRefsTimeout=10000 -c core.filesRefLockTimeout=1000 symbolic-ref --end-of-options HEAD]: fork/exec /var/opt/gitlab/gitaly/run/gitaly-565/git-exec-484403296.d/git: resource temporarily unavailable. (Gitlab::Git::CommandError)
        from /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/git/wraps_gitaly_errors.rb:23:in `handle_error'
        from /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/git/wraps_gitaly_errors.rb:14:in `rescue in wrapped_gitaly_errors'
        from /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/git/wraps_gitaly_errors.rb:6:in `wrapped_gitaly_errors'
        from /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/git/repository.rb:103:in `root_ref'
        from /opt/gitlab/embedded/service/gitlab-rails/app/models/repository.rb:620:in `root_ref'
        from /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/repository_cache_adapter.rb:107:in `block (2 levels) in cache_method_asymmetrically'
        from /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/repository_cache.rb:44:in `fetch_without_caching_false'
        from /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/repository_cache_adapter.rb:202:in `block (2 levels) in cache_method_output_asymmetrically'
        from /opt/gitlab/embedded/lib/ruby/gems/3.3.0/gems/request_store-1.7.0/lib/request_store.rb:51:in `fetch'
        from /opt/gitlab/embedded/lib/ruby/gems/3.3.0/gems/forwardable-1.3.3/lib/forwardable.rb:240:in `fetch'
        from /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/repository_cache.rb:25:in `fetch'

These errors indicate potential resource exhaustion or thread scheduling issues.

Process monitoring:

  • pids.current showed stable overnight baseline (~1040 of 2048 limit) with no leak pattern, suggesting the issue is load-driven rather than a resource leak.

Root Cause Analysis (Suspected)

The issue appears to be related to changes in Ruby 3.3's thread scheduling behavior. Ruby 3.3 introduced refactoring for M:N thread support (Ruby Bug - https://bugs.ruby-lang.org/issues/20816), which changed how threads are scheduled when releasing the GVL (Global VM Lock).

Hypothesis: Under high Sidekiq concurrency, the heartbeat thread may be experiencing scheduling delays that prevent it from refreshing its Redis key within the 60-second TTL, causing the process to silently deregister.

Evidence Supporting Thread Scheduling Hypothesis

  1. Concurrency correlation: Reducing sidekiq['concurrency'] from 20 to 5 immediately resolved the deregistration, suggesting GVL contention may be a factor.
  2. Reproducible across instances: Both affected instances upgraded from 18.8 to 18.9 at the same time with no other configuration or infrastructure changes.
  3. Version-specific: Issue only appears after Ruby 3.3 upgrade; no issues reported on 18.8 with Ruby 3.2.x.

Output of checks

Environment:

  • GitLab 18.9.1 EE, Omnibus
  • Deployment: Podman Quadlet
  • Ruby: 3.3.10
  • Original configuration: sidekiq['queue_groups'] = ['*'] * 2 (2 processes, 20 threads each)

Workaround configuration:

  • sidekiq['concurrency'] = 5 (stable for now)
  • Customer exploring horizontal scaling: 8 processes × 5 threads
sidekiq['queue_groups'] = ['*'] * 8 
sidekiq['concurrency'] = 5

Additional Context

Related Issues:

Impact:

  • Complete Sidekiq outage under normal daytime load
  • All Sidekiq-dependent functionality stops (CI pipelines, merge request updates, emails, webhooks, etc.)
  • Requires manual restart to recover
  • Potentially affects any self-managed instance running 18.9+ with moderate-to-high Sidekiq concurrency

Questions for Investigation:

  • Is it possible to consider staying on Ruby 3.2.x or reverting to it in future GitLab versions if Ruby 3.3 continues to exhibit thread scheduling issues with Sidekiq?
Edited by 🤖 GitLab Bot 🤖