High memory usage in project export
I've noticed a simple project import/export of my internal GitLab CE instance getting killed by the Sidekiq RSS killer after consuming 1.2+ GB RAM. Wondering why, I took an Object Space dump by doing ProjectExportWorker.new.perform(user.id, project.id)
in a Rails console:
Analyzing Heap (Generation: all)
-------------------------------
allocated by memory (319827013) (in bytes)
==============================
32117832 /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/ci/config/entry/factory.rb:63
28904960 /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/ci/config/entry/node.rb:17
19789982 /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/activesupport-4.2.8/lib/active_support/json/encoding.rb:87
17008840 /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/ci/config/entry/node.rb:19
15515920 /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/ci/config/entry/job.rb:100
15361256 /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/activesupport-4.2.8/lib/active_support/core_ext/object/json.rb:159
14236216 /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/ci/config/entry/image.rb:38
14194394 /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/ci/config/entry/job.rb:104
13398376 /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/activesupport-4.2.8/lib/active_support/core_ext/hash/keys.rb:142
13080810 /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/activerecord-4.2.8/lib/active_record/type/string.rb:35
11164352 /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/activesupport-4.2.8/lib/active_support/json/encoding.rb:91
11156200 /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/activesupport-4.2.8/lib/active_support/callbacks.rb:446
9549496 /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/ci/config/entry/variables.rb:20
9448920 /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/activemodel-4.2.8/lib/active_model/errors.rb:73
9448920 /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/activemodel-4.2.8/lib/active_model/validations.rb:301
8210752 /opt/gitlab/embedded/lib/ruby/2.3.0/psych/visitors/to_ruby.rb:162
7329904 /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/activerecord-4.2.8/lib/active_record/result.rb:116
5879888 /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/ci/config/entry/node.rb:61
4264120 /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/ci/config/entry/undefined.rb:10
4264120 /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/ci/config/entry/factory.rb:40
4164205 /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/activerecord-4.2.8/lib/active_record/connection_adapters/postgresql/database_statements.rb:168
4014520 /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/activerecord-4.2.8/lib/active_record/result.rb:110
3824372 /opt/gitlab/embedded/lib/ruby/2.3.0/psych.rb:379
3651848 /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/activesupport-4.2.8/lib/active_support/core_ext/hash/keys.rb:146
3527392 /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/activerecord-4.2.8/lib/active_record/attribute.rb:5
3439472 /opt/gitlab/embedded/lib/ruby/site_ruby/2.3.0/rbreadline.rb:4381
2933424 /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/activerecord-4.2.8/lib/active_record/attribute_set/builder.rb:32
2132544 /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/activemodel-4.2.8/lib/active_model/serialization.rb:107
1917497 /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/ci/config/entry/jobs.rb:38
1864933 /opt/gitlab/embedded/service/gitlab-rails/app/models/blob.rb:93
1563681 /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/activemodel-4.2.8/lib/active_model/attribute_methods.rb:383
1473600 /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/activesupport-4.2.8/lib/active_support/core_ext/time/calculations.rb:37
1408784 /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/activesupport-4.2.8/lib/active_support/core_ext/hash/except.rb:10
1340182 /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/activerecord-4.2.8/lib/active_record/connection_adapters/postgresql/database_statements.rb:162
1196728 /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/activerecord-4.2.8/lib/active_record/querying.rb:40
887850 /opt/gitlab/embedded/lib/ruby/2.3.0/psych/scalar_scanner.rb:41
832520 /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/ci/config/entry/services.rb:31
625928 /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/activerecord-4.2.8/lib/active_record/relation.rb:34
614000 /opt/gitlab/embedded/lib/ruby/2.3.0/time.rb:676
600040 /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/encoding_helper.rb:43
I think the bulk of these files have to do with .gitlab-ci.yml
parsing, but I'm quite surprised to see them this prominently in the project exporter.
Proposal
Reduce amount of N+1 queries execute and make the memory much easier to recycle.
This is solved by: https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/32704, https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/32423.
Edited by 🤖 GitLab Bot 🤖