Allow execution and memory profiling of Sidekiq jobs
Problem to solve
As per https://gitlab.com/gitlab-org/gitlab-ce/issues/62270#future-extensions, allow execution and memory profiling of Sidekiq jobs.
Intended users
Proposal
Proposal 1: set profile option(memory/execution, Worker) through HTTP Header from Web UI.
Cons: only works for tasks trigger through GUI, like project import
, project export
.
- use HTTP Header through
admin area
, same as how we usestack-profile
andmemory-profile
today(https://docs.gitlab.com/ee/administration/monitoring/performance/request_profiling.html).- extend the mode
HTTP_X_PROFILE_MODE
to have two new values:sidekiq_execution
,sidekiq_memory
- add a new header
HTTP_X_PROFILE_SIDEKIQ_WORKER
to tell which Worker to profile (this can be extended in future) - Limitation: it only works for tasks trigger through GUI, like
project import
,project export
.
- extend the mode
- use SafeRequestStore to pass the profile options from controller to WorkerClass
- in Worker perform_async/bulk_perform_async/bulk_perform_in, write sidekiq profile options to Redis
- Add sidekiq middleware to run profile, based on sidekiq profile options for the JID, from Redis
Proposal 2: create a new section in Admin page to set profile option(memory/execution, Worker).
Cons: can not be accurate to Job level, only Worker level.
- set profile mode(execution/memory) and Worker to profile in admin page.
- do the same as step 4 in Proposal 1. But it will delete the Redis key, so it only run profile on the first job of the specified Worker.
https://github.com/callumj/sidekiq_profiling_middleware
Proposal 3: use this toolCons: need to change source code(sidekiq configure_server)
- TBD: this tool is not actively maintained. Are we ok using it?
- TBD: this tool has limitation. Feels like setting up in production is tricky. If we use it locally in GDK it may be ok.
you will need to set your Sidekiq concurrency to 1. This will ensure that only one job operates at a time ensuring other jobs don't confuse your memory profile or stacktraces https://github.com/callumj/sidekiq_profiling_middleware#concurrency-set-it-to-1
Links / references
https://github.com/callumj/sidekiq_profiling_middleware
https://docs.gitlab.com/ee/administration/monitoring/performance/request_profiling.html
Edited by Qingyu Zhao