Investigate ActionCable memory spikes in prod

Problem

While rolling out ActionCable for real time issue assignees, we observed that Puma worker memory started spiking as the number of connections went up, eventually exceeding the thresholds we define as healthy.

This is currently blocking this feature from going live, and the feature has been disabled meanwhile.

We need to investigate what is causing this and how to remedy it.

Original comment:

I'm seeing high memory utilization on the pods servicing these requests

https://dashboards.gitlab.net/d/git-pod/git-pod-info?orgId=1&from=now-6h&to=now

It looks like the memory consumption is hitting our limit in under < 1 hour

Support data

Memory and AC connections during the attempted rollout:

https://thanos-query.ops.gitlab.net/new/graph?g0.expr=avg%20by%20(pid)%20(ruby_gc_stat_heap_live_slots%7Btype%3D%22git%22%2C%20env%3D%22gprd%22%2C%20app%3D%22webservice%22%7D)&g0.tab=0&g0.stacked=0&g0.range_input=12h&g0.max_source_resolution=0s&g0.deduplicate=1&g0.partial_response=0&g0.end_input=2020-10-05%2018%3A07%3A56&g0.moment_input=2020-10-05%2018%3A07%3A56&g1.expr=max%20by%20(pid)%20(action_cable_active_connections%7Benv%3D%22gprd%22%2C%20type%3D%22git%22%2C%20stage%3D%22main%22%2C%20app%3D%22webservice%22%7D)&g1.tab=0&g1.stacked=0&g1.range_input=12h&g1.max_source_resolution=0s&g1.deduplicate=1&g1.partial_response=0&g1.end_input=2020-10-05%2018%3A07%3A56&g1.moment_input=2020-10-05%2018%3A07%3A56&g2.expr=avg%20by%20(pid)%20(ruby_gc_stat_heap_allocated_pages%7Btype%3D%22git%22%2C%20env%3D%22gprd%22%2C%20app%3D%22webservice%22%7D)&g2.tab=0&g2.stacked=0&g2.range_input=12h&g2.max_source_resolution=0s&g2.deduplicate=1&g2.partial_response=0&g2.end_input=2020-10-05%2017%3A14%3A34&g2.moment_input=2020-10-05%2017%3A14%3A34&g3.expr=max%20by%20(pid)%20(ruby_process_resident_memory_bytes%7Benv%3D%22gprd%22%2C%20type%3D%22git%22%2C%20stage%3D%22main%22%2C%20app%3D%22webservice%22%7D)&g3.tab=0&g3.stacked=0&g3.range_input=12h&g3.max_source_resolution=0s&g3.deduplicate=1&g3.partial_response=0&g3.end_input=2020-10-05%2018%3A07%3A56&g3.moment_input=2020-10-05%2018%3A07%3A56&g4.expr=avg%20by%20(pid)%20(rate(ruby_gc_stat_total_allocated_objects%7Benv%3D%22gprd%22%2C%20type%3D%22git%22%2C%20app%3D%22webservice%22%7D%5B5m%5D))&g4.tab=0&g4.stacked=0&g4.range_input=12h&g4.max_source_resolution=0s&g4.deduplicate=1&g4.partial_response=0&g4.end_input=2020-10-05%2018%3A07%3A56&g4.moment_input=2020-10-05%2018%3A07%3A56