Default Puma per_worker_max_memory_mb is too low

On a 2-core 8 GB RAM Omnibus instance, I'm seeing regular warnings of memory limits exceeded:

{"timestamp":"2020-10-14T05:31:33.755Z","pid":30780,"message":"- Worker 1 (pid: 15094) booted, phase: 0"}
{"timestamp":"2020-10-14T05:31:44.515Z","pid":30780,"message":"PumaWorkerKiller: Out of memory. 2 workers consuming total: 2355.65234375 mb out of max: 2205.0 mb. Sending TERM to pid 14981 consuming 788.26953125 mb."}
{"timestamp":"2020-10-14T05:31:54.061Z","pid":30780,"message":"- Worker 0 (pid: 15200) booted, phase: 0"}
{"timestamp":"2020-10-14T05:32:04.517Z","pid":30780,"message":"PumaWorkerKiller: Out of memory. 2 workers consuming total: 2348.49609375 mb out of max: 2205.0 mb. Sending TERM to pid 15094 consuming 781.07421875 mb."}
{"timestamp":"2020-10-14T05:32:14.401Z","pid":30780,"message":"- Worker 1 (pid: 15470) booted, phase: 0"}
{"timestamp":"2020-10-14T05:32:24.518Z","pid":30780,"message":"PumaWorkerKiller: Out of memory. 2 workers consuming total: 2349.6484375 mb out of max: 2205.0 mb. Sending TERM to pid 15200 consuming 782.2578125 mb."}
{"timestamp":"2020-10-14T05:32:34.728Z","pid":30780,"message":"- Worker 0 (pid: 15580) booted, phase: 0"}
{"timestamp":"2020-10-14T05:32:44.519Z","pid":30780,"message":"PumaWorkerKiller: Out of memory. 2 workers consuming total: 2348.59375 mb out of max: 2205.0 mb. Sending TERM to pid 15580 consuming 781.18359375 mb."}

The calculation comes from https://gitlab.com/gitlab-org/gitlab/blob/fcea6c498dbaeaeb11fa51377e541cdb6ad7ba42/lib/gitlab/cluster/puma_worker_killer_initializer.rb#L8-26. For a 2-worker node using min and max threads of 4, we have max memory limit of 2205 MB:

550 + (850 * 2) = 2250

However, on my instance, I am seeing both the master and worker processes use about 800 MB, which consumes 2400 MB. This puts us over the limit in every request.

I suggest we raise these default limits while investigating how we might lower the baseline memory usage. On GitLab.com, we use these defaults:

puma['worker_processes'] = 16
puma['worker_timeout'] = 60
puma['min_threads'] = 1
puma['max_threads'] = 4
puma['per_worker_max_memory_mb'] = 1342