Skip to content

Prevent Sidekiq size limiter middleware from running multiple times on the same job

Heinrich Lee Yu requested to merge improve-size-limiter-middleware into master

What does this MR do and why?

In gitlab-com/gl-infra/scalability#1179 (comment 705409681), we saw that this middleware is slow.

Gitlab::CurrentSettings first checks the request store then loads it from the process memory cache or the DB if it's not available.

In the Sidekiq scheduler thread, request store is not available so we always retrieve from memory cache. And we do this 3 times per job in the constructor. We can see that this isn't very fast:

[6] pry(main)> ApplicationSetting.current
=> ...
[7] pry(main)> puts Benchmark.measure { 3000.times { ApplicationSetting.cache_backend.read(ApplicationSetting.cache_key) } }
  3.423157   0.071603   3.494760 (  3.494570)

This MR prevents the size validator from running twice. This means this middleware will not be run by the scheduler thread.

How to set up and validate locally

  1. Enqueue many scheduled jobs:

    Sidekiq::Client.push_bulk('class' => PipelineScheduleWorker, 'args' => 1000.times.map { [] }, 'at' => 5.second.from_now.to_i)

  2. Monitor the Sidekiq logs and find the "Enqueuing scheduled jobs" entries

Before these changes:

{"severity":"INFO","time":"2021-10-16T07:05:46.842Z","message":"Enqueuing scheduled jobs","status":"done","sorted_set":"schedule","jobs_count":1000,"redundant_jobs_count":0,"duration_s":5.827877908013761,"retry":0}

With the validated? check:

{"severity":"INFO","time":"2021-10-16T11:38:30.484Z","message":"Enqueuing scheduled jobs","status":"done","sorted_set":"schedule","jobs_count":1000,"redundant_jobs_count":0,"duration_s":2.3445153500069864,"retry":0}

Related to gitlab-com/gl-infra/scalability#1179 (closed) / #342092 (closed)

MR acceptance checklist

This checklist encourages us to confirm any changes have been analyzed to reduce risks in quality, performance, reliability, security, and maintainability.

Edited by Heinrich Lee Yu

Merge request reports