Skip to content
GitLab
Next
Closed
Issue created by Stan Hu@stanhuOwner

High memory usage in project export

I've noticed a simple project import/export of my internal GitLab CE instance getting killed by the Sidekiq RSS killer after consuming 1.2+ GB RAM. Wondering why, I took an Object Space dump by doing ProjectExportWorker.new.perform(user.id, project.id) in a Rails console:

Analyzing Heap (Generation: all)
-------------------------------

allocated by memory (319827013) (in bytes)
==============================
  32117832  /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/ci/config/entry/factory.rb:63
  28904960  /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/ci/config/entry/node.rb:17
  19789982  /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/activesupport-4.2.8/lib/active_support/json/encoding.rb:87
  17008840  /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/ci/config/entry/node.rb:19
  15515920  /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/ci/config/entry/job.rb:100
  15361256  /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/activesupport-4.2.8/lib/active_support/core_ext/object/json.rb:159
  14236216  /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/ci/config/entry/image.rb:38
  14194394  /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/ci/config/entry/job.rb:104
  13398376  /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/activesupport-4.2.8/lib/active_support/core_ext/hash/keys.rb:142
  13080810  /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/activerecord-4.2.8/lib/active_record/type/string.rb:35
  11164352  /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/activesupport-4.2.8/lib/active_support/json/encoding.rb:91
  11156200  /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/activesupport-4.2.8/lib/active_support/callbacks.rb:446
   9549496  /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/ci/config/entry/variables.rb:20
   9448920  /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/activemodel-4.2.8/lib/active_model/errors.rb:73
   9448920  /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/activemodel-4.2.8/lib/active_model/validations.rb:301
   8210752  /opt/gitlab/embedded/lib/ruby/2.3.0/psych/visitors/to_ruby.rb:162
   7329904  /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/activerecord-4.2.8/lib/active_record/result.rb:116
   5879888  /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/ci/config/entry/node.rb:61
   4264120  /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/ci/config/entry/undefined.rb:10
   4264120  /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/ci/config/entry/factory.rb:40
   4164205  /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/activerecord-4.2.8/lib/active_record/connection_adapters/postgresql/database_statements.rb:168
   4014520  /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/activerecord-4.2.8/lib/active_record/result.rb:110
   3824372  /opt/gitlab/embedded/lib/ruby/2.3.0/psych.rb:379
   3651848  /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/activesupport-4.2.8/lib/active_support/core_ext/hash/keys.rb:146
   3527392  /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/activerecord-4.2.8/lib/active_record/attribute.rb:5
   3439472  /opt/gitlab/embedded/lib/ruby/site_ruby/2.3.0/rbreadline.rb:4381
   2933424  /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/activerecord-4.2.8/lib/active_record/attribute_set/builder.rb:32
   2132544  /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/activemodel-4.2.8/lib/active_model/serialization.rb:107
   1917497  /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/ci/config/entry/jobs.rb:38
   1864933  /opt/gitlab/embedded/service/gitlab-rails/app/models/blob.rb:93
   1563681  /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/activemodel-4.2.8/lib/active_model/attribute_methods.rb:383
   1473600  /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/activesupport-4.2.8/lib/active_support/core_ext/time/calculations.rb:37
   1408784  /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/activesupport-4.2.8/lib/active_support/core_ext/hash/except.rb:10
   1340182  /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/activerecord-4.2.8/lib/active_record/connection_adapters/postgresql/database_statements.rb:162
   1196728  /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/activerecord-4.2.8/lib/active_record/querying.rb:40
    887850  /opt/gitlab/embedded/lib/ruby/2.3.0/psych/scalar_scanner.rb:41
    832520  /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/ci/config/entry/services.rb:31
    625928  /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/activerecord-4.2.8/lib/active_record/relation.rb:34
    614000  /opt/gitlab/embedded/lib/ruby/2.3.0/time.rb:676
    600040  /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/encoding_helper.rb:43

I think the bulk of these files have to do with .gitlab-ci.yml parsing, but I'm quite surprised to see them this prominently in the project exporter.

project_export.txt

Proposal

Reduce amount of N+1 queries execute and make the memory much easier to recycle.

This is solved by: https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/32704, https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/32423.

Edited by 🤖 GitLab Bot 🤖