API processes hitting memory limits and restarting
The Gitter API processes are hitting our Monit memory thresholds/limits and restarting.
Example PagerDuty alerts:
- https://gitter.pagerduty.com/incidents/PPD92JT
- https://gitter.pagerduty.com/incidents/PBB0DXH
- https://gitter.pagerduty.com/incidents/P6J5EKV
webapp-02: Profiling CPU usage /tmp/tmp.fLIgjDrYWl.deploy-tools-profile / kernel.kptr_restrict = 0 kernel.perf_event_paranoid = 0 / profile-service-cpu failed Profiling memory usage /var/log/gitter/heap.webapp-02.gitter-api-2.22938.2020-04-01-093214.heapsnapshot S3 upload failed
| Service | Monit Status | Monit Service | Monit Details | Host Info | Host |
|---|---|---|---|---|---|
| gitter-api-2_terminated | http://webapp-02:2812/gitter-api-2 | gitter-api-2 | mem amount of 948.7 MB matches resource limit [mem amount<900.0 MB] | https://app.datadog.com | webapp-02 |
Questions
- Is this natural growth?
- What's taking up all of the memory?
cc @viktomas