I've noticed a simple project import/export of my internal GitLab CE instance getting killed by the Sidekiq RSS killer after consuming 1.2+ GB RAM. Wondering why, I took an [Object Space dump](https://blog.codeship.com/the-definitive-guide-to-ruby-heap-dumps-part-i/) by doing `ProjectExportWorker.new.perform(user.id, project.id)` in a Rails console: ``` Analyzing Heap (Generation: all) ------------------------------- allocated by memory (319827013) (in bytes) ============================== 32117832 /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/ci/config/entry/factory.rb:63 28904960 /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/ci/config/entry/node.rb:17 19789982 /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/activesupport-4.2.8/lib/active_support/json/encoding.rb:87 17008840 /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/ci/config/entry/node.rb:19 15515920 /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/ci/config/entry/job.rb:100 15361256 /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/activesupport-4.2.8/lib/active_support/core_ext/object/json.rb:159 14236216 /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/ci/config/entry/image.rb:38 14194394 /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/ci/config/entry/job.rb:104 13398376 /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/activesupport-4.2.8/lib/active_support/core_ext/hash/keys.rb:142 13080810 /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/activerecord-4.2.8/lib/active_record/type/string.rb:35 11164352 /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/activesupport-4.2.8/lib/active_support/json/encoding.rb:91 11156200 /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/activesupport-4.2.8/lib/active_support/callbacks.rb:446 9549496 /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/ci/config/entry/variables.rb:20 9448920 /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/activemodel-4.2.8/lib/active_model/errors.rb:73 9448920 /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/activemodel-4.2.8/lib/active_model/validations.rb:301 8210752 /opt/gitlab/embedded/lib/ruby/2.3.0/psych/visitors/to_ruby.rb:162 7329904 /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/activerecord-4.2.8/lib/active_record/result.rb:116 5879888 /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/ci/config/entry/node.rb:61 4264120 /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/ci/config/entry/undefined.rb:10 4264120 /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/ci/config/entry/factory.rb:40 4164205 /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/activerecord-4.2.8/lib/active_record/connection_adapters/postgresql/database_statements.rb:168 4014520 /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/activerecord-4.2.8/lib/active_record/result.rb:110 3824372 /opt/gitlab/embedded/lib/ruby/2.3.0/psych.rb:379 3651848 /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/activesupport-4.2.8/lib/active_support/core_ext/hash/keys.rb:146 3527392 /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/activerecord-4.2.8/lib/active_record/attribute.rb:5 3439472 /opt/gitlab/embedded/lib/ruby/site_ruby/2.3.0/rbreadline.rb:4381 2933424 /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/activerecord-4.2.8/lib/active_record/attribute_set/builder.rb:32 2132544 /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/activemodel-4.2.8/lib/active_model/serialization.rb:107 1917497 /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/ci/config/entry/jobs.rb:38 1864933 /opt/gitlab/embedded/service/gitlab-rails/app/models/blob.rb:93 1563681 /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/activemodel-4.2.8/lib/active_model/attribute_methods.rb:383 1473600 /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/activesupport-4.2.8/lib/active_support/core_ext/time/calculations.rb:37 1408784 /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/activesupport-4.2.8/lib/active_support/core_ext/hash/except.rb:10 1340182 /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/activerecord-4.2.8/lib/active_record/connection_adapters/postgresql/database_statements.rb:162 1196728 /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/activerecord-4.2.8/lib/active_record/querying.rb:40 887850 /opt/gitlab/embedded/lib/ruby/2.3.0/psych/scalar_scanner.rb:41 832520 /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/ci/config/entry/services.rb:31 625928 /opt/gitlab/embedded/lib/ruby/gems/2.3.0/gems/activerecord-4.2.8/lib/active_record/relation.rb:34 614000 /opt/gitlab/embedded/lib/ruby/2.3.0/time.rb:676 600040 /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/encoding_helper.rb:43 ``` I think the bulk of these files have to do with `.gitlab-ci.yml` parsing, but I'm quite surprised to see them this prominently in the project exporter. [project_export.txt](/uploads/8ae61dba89700b24e8764abc7a17a9bf/project_export.txt) ## Proposal Reduce amount of N+1 queries execute and make the memory much easier to recycle. This is solved by: https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/32704, https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/32423.