Reducing memory usage of Sidekiq tasks
This issue seeks to profile and discuss possible ways to reduce memory usage of Sidekiq tasks. ZenDesk issue: https://gitlab.zendesk.com/agent/tickets/28848.
Attached is the Sidekiq memory profile output using this gem after pushing 673 commits from 8.9 RC1 to RC8 with an open MR.
High-level stats:
Total allocated: 1532771872 bytes (3674543 objects)
Total retained: 8712646 bytes (18922 objects)
allocated memory by gem
-----------------------------------
1304777487 embedded/lib
131201999 activerecord-4.2.6
34888039 gitlab_git-10.2.3
18294371 activesupport-4.2.6
11421226 arel-6.0.3
6359276 activemodel-4.2.6
5800902 gitlab-rails/lib
5308953 redis-3.3.0
4948654 nokogiri-1.6.8
4343896 gitlab-rails/app
1372880 other
970972 rufus-scheduler-3.1.10
539692 actionpack-4.2.6
481434 redis-namespace-1.5.2
437424 paranoia-2.1.4
371517 rack-1.6.4
318740 redis-store-1.1.7
297196 connection_pool-2.2.0
204832 redis-activesupport-4.1.5
77430 acts-as-taggable-on-3.5.0
60059 json-1.8.3
55196 sidekiq-4.1.2
55060 i18n-0.7.0
49353 state_machines-0.4.0
36612 sidekiq-cron-0.4.0
15824 rugged-0.24.0
15032 html-pipeline-1.11.0
11128 kaminari-0.17.0
8692 state_machines-activemodel-0.4.0
8450 thread_safe-0.3.5
7847 carrierwave-0.10.0
7640 sanitize-2.1.0
6372 state_machines-activerecord-0.4.0
6031 pg-0.18.4
4440 tzinfo-1.2.2
2496 attr_encrypted-3.0.1
1888 elasticsearch-model-0.1.8
1824 default_value_for-3.0.1
968 concurrent-ruby-1.0.2
40 after_commit_queue-1.3.0
I'm not sure if I understand why the bulk of the memory seems to be taken by something in the Timeout
class:
allocated memory by file
-----------------------------------
1184661200 /opt/gitlab/embedded/lib/ruby/2.1.0/timeout.rb
45590734 /opt/gitlab/embedded/lib/ruby/2.1.0/psych/visitors/yaml_tree.rb
39803445 /opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/activerecord-4.2.6/lib/active_record/connection_adapters/postgresql_adapter.rb
36118171 /opt/gitlab/embedded/lib/ruby/2.1.0/psych/scalar_scanner.rb
26668819 /opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/gitlab_git-10.2.3/lib/gitlab_git/commit.rb
20125587 /opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/activerecord-4.2.6/lib/active_record/connection_adapters/postgresql/database_statements.rb
14137696 /opt/gitlab/embedded/lib/ruby/2.1.0/psych/tree_builder.rb
Edited by 🤖 GitLab Bot 🤖