KR: Support observability & resolve known memory issues => 80%%

• Resolve known memory issues
  • Investigate ways to resolve memory pressure on Workhorse pods supporting WebSockets gitlab-org/gitlab-workhorse#318 (comment 502588669)
  • Improve memory reporting by switching to MADV_DONTNEEDfrom MADV_FREE strategy for releasing memory gitlab-org/gitlab-workhorse#352 (closed)
  • Attempt to reduce buffer size to ~4kB based on outcome from tracking message size (non blocker)
• Implement Prometheus metrics in support of observability
  • Support Rails Action Cable instrumentation gitlab-org/gitlab#296845 (closed)
  • Track message size sent over WebSocket connections gitlab-org/gitlab#326364 (closed)
• Address scalability concerns, unblocking further roll-out
  • Establish whether current cost is sustainable
  • Reliably estimate a cost of further rollout
  • Produce a mechanism for other teams to achieve scalability for new features