Remove github-linguist dependency from either unicorn or Sidekiq
Using derailed:benchmarks
, it seems that github-linguist uses 100 MB out of 224 MB RAM in our process (full profile attached below):
TOP: 224.4258 MiB
linguist: 99.9883 MiB
linguist/language: 82.8281 MiB (Also required by: linguist/lazy_blob)
linguist/blob_helper: 16.6172 MiB (Also required by: linguist/file_blob, linguist/blob, and 2 others)
mime/types: 15.3008 MiB (Also required by: mime/types/columnar, /Users/stanhu/.rbenv/versions/2.3.3/lib/ruby/gems/2.3.0/gems/rest-client-2.0.0/lib/restclient/request)
charlock_holmes: 0.7773 MiB (Also required by: TOP)
charlock_holmes/charlock_holmes: 0.3906 MiB
charlock_holmes/encoding_detector: 0.3438 MiB
I think we mostly use this gem to calculate our language graphs, and appears we do this synchronously in Unicorn. However, it seems like a giant footprint to force upon all unicorn workers. I wonder if we can somehow remove this either from Sidekiq or unicorn, which would would free up 100 MB/process.
I know Gitaly was looking at putting this into the sidecar, and perhaps we should prioritize that to free up a lot of RAM per worker.
Edited by Stan Hu