Investigate high memory use in redis connections

Everyone can contribute. Help move this issue forward while earning points, leveling up and collecting rewards.

• Close this issue

The following discussion from !91853 (merged) should be addressed:

• @engwan started a discussion: (+2 comments)

  This is now in staging canary so I compared a request with canary enabled and disabled.

  https://staging.gitlab.com/gitlab-org/gitlab has 45k branches so I tested an issue in this project.

  gstg-cny: https://nonprod-log.gitlab.net/app/discover#/doc/ed942d00-5186-11ea-ad8a-f3610a492295/pubsub-rails-inf-gstg-000660?id=AB9g-4EBYWFSUzBEiJQf

  gstg: https://nonprod-log.gitlab.net/app/discover#/doc/ed942d00-5186-11ea-ad8a-f3610a492295/pubsub-rails-inf-gstg-000660?id=Te9h-4EBPVFN7vZl0ECf

  duration_s is down from 2.937 to 0.175. mem_bytes is down from 1.2g to 6mb.

  I'm still confused why this uses that much memory even in the old code. redis_cache_read_bytes is "just" 1.6mb. Is the Ruby socket library that inefficient? Or is it the allocation of 45k+ string objects after we get that from the Redis set? 🤔

  Maybe it's worth taking another look at hiredis: !25850 (closed). The problem experienced there was also large SMEMBERS results. So I wonder if it also reduces memory consumption.

  I'll try to take a look when I have time.