Allow execution and memory profiling of Sidekiq jobs
Problem to solve
Qingyu: when I quickly read this issue today, I feel we really need to re-think what is the usage scenario of this tool. Some quick questions for myself:
- environment: local GDK? staging? production?
- how to trigger: turn on/off through admin UI? or through ENV variable when server initialise.
- how to specify which job to profile: is it easy to specify one job? how?
- May need to research other options/tools: as mentioned below, some existing tool(https://github.com/callumj/sidekiq_profiling_middleware) has limitation, it requires set Sidekiq concurrency to 1.
Below are old issue description. The original idea was: we want to profile Sidekiq jobs using the same way as existing request_profile. But old issue did not dive into details whether it is possible/easy. While started working on, we found some technical difficulties, as posted in the discussions.
As per https://gitlab.com/gitlab-org/gitlab-ce/issues/62270#future-extensions, allow execution and memory profiling of Sidekiq jobs.
Admin can trigger the profile through
Admin Area->Monitoring->Background Jobs(TBD: where/how exactly to add.)
Maybe can use some tool such as: https://github.com/callumj/sidekiq_profiling_middleware
- TBD: this tool is not actively maintained. Are we ok using it?
- TBD: this tool has limitation. Feels like setting up in production is tricky. If we use it locally in GDK it may be ok.
you will need to set your Sidekiq concurrency to 1. This will ensure that only one job operates at a time ensuring other jobs don't confuse your memory profile or stacktraces https://github.com/callumj/sidekiq_profiling_middleware#concurrency-set-it-to-1
What does success look like, and how can we measure that?
User can select the job from
Admin Area->Monitoring->Background Jobs, select profile mode(
memory), and trigger the job(Question: can admin put a job in higher priority in sidekiq?). After the job finished, profile result/report will be listed at
Admin Area->Monitoring->Background Jobs page. This similar to
Admin Area->Monitoring->Requests Profiles