<!-- copied from https://gitlab.com/gitlab-org/gitlab-ce/blob/master/.gitlab/issue_templates/Feature%20proposal.md --> ### Problem to solve We've noticed before that Puma can see climbing memory use in production over a period of days, especially during weekends, where there are no deploys that would reset these gauges. Meanwhile, we are getting an increased number of customer reports complaining that the Puma memory killer frequently kills Puma processes, which causes throughput issues. We should see if we can find out the reason for any potential runaway memory issues, be it just Ruby heap fragmentation or a memory leak. The issue does not seem as pronounced on Sidekiq, where (across all fleets), memory seems to plateau at some point:  [Source](https://thanos-query.ops.gitlab.net/graph?g0.expr=avg%20by%20(app)%20(ruby_process_resident_memory_bytes%7Benv%3D%22gprd%22%7D)&g0.tab=0&g0.stacked=0&g0.range_input=1w&g0.max_source_resolution=0s&g0.deduplicate=1&g0.partial_response=0&g0.store_matches=%5B%5D&g0.end_input=2022-04-28%2000%3A00%3A00&g0.moment_input=2022-04-28%2000%3A00%3A00) On top of that actual issue, it was [pointed out](https://gitlab.com/groups/gitlab-org/-/epics/8105#note_952415703) that having to rely on SREs to pull diagnostic data for ~"group::memory" to analyze, we should provide a framework that automates performance data collection and publishes it in a place that product developers have access to. ### Proposal We can break down the work into two steps: 1. **A mechanism to signal the application to produce diagnostic reports.** These will remain in pod-local storage for now. This still requires help from an SRE to pull down these files, but this could be automated with a script. 1. **A mechanism to put these files into object storage.** To completely remove the necessity for SRE support, we publish these reports into GCS instead, where developers can download them for further analysis. ### Links / references - https://gitlab.com/gitlab-org/gitlab/-/issues/334831