Large scheduling delay in Sidekiq urgent-cpu-bound
@leetickett noticed it took a while for his merge request to update in gitlab-org/gitlab!37748 (merged). In Kibana, I see sidekiq_scheduling_latency_s
took 212 s for UpdateMergeRequestsWorker
to start running (https://log.gprd.gitlab.net/goto/e4e2d155ca8718d259be2f71d8de0b27). I'm not sure what's going on, but I'm filing it in case we start to see a pattern:
If we look over a 3 period for urgent-cpu-bound
(https://log.gprd.gitlab.net/goto/5c02e4579f5fb07a9c3742595c2c442d) for the max, we see these spikes:
https://log.gprd.gitlab.net/goto/b9ffa6ca5d7f9df1cbfdc5907452da9e shows the P99 value is over 10 seconds over the last 24 hours: