High memory consumtion of gitlab puma workers
Summary
After upgrading our self hosted Gitlab-CE from version 16.0.1 to 16.2.1 downloading artifacts fills up the system memory.
Adjusting any settings described in the documentation doesn't make any mayor difference Puma Documentation
Previously with 16.0.1 we didn't have any issue with memory consumption with the same sized artifacts and frequent downloads of said artifacts.
When looking through old and new application_json.log files i found that the following errors always occur after updating the system These logs are also occurring every time we download an artifact
{"severity":"WARN","time":"2023-08-02T09:46:09.732Z","memwd_max_strikes":5,"memwd_cur_strikes":6,"message":"rss memory limit exceeded","memwd_rss_bytes":1476722688,"memwd_max_rss_bytes":1258291200,"memwd_handler_class":"Gitlab::Memory::Watchdog::Handlers::PumaHandler","memwd_sleep_time_s":60,"pid":26177,"worker_id":"puma_0"}
{"severity":"INFO","time":"2023-08-02T09:46:09.740Z","message":"stopped","memwd_reason":"successfully handled","memwd_handler_class":"Gitlab::Memory::Watchdog::Handlers::PumaHandler","memwd_sleep_time_s":60,"pid":26177,"worker_id":"puma_0","memwd_rss_bytes":1476722688}
{"severity":"DEBUG","time":"2023-08-02T09:46:10.184Z","message":"ActiveRecord connection established"}
{"severity":"INFO","time":"2023-08-02T09:46:10.228Z","message":"started","memwd_handler_class":"Gitlab::Memory::Watchdog::Handlers::PumaHandler","memwd_sleep_time_s":60,"pid":28320,"worker_id":"puma_0","memwd_rss_bytes":749039616}
What is the current bug behavior?
With version 16.2.1:
- Clicking download an artifact takes up to 30 seconds before the download starts
- Downloading artifacts of the size of ~2GB fills the memory for a few minutes (circa 3 minutes)
- If multiple different artifacts are downloaded and the memory is being capped the whole server takes a dump and gitlab is unresponsive until the puma cluster workers are freeing the memory again
What is the expected correct behavior?
With version 16.0.1:
- Clicking download instantly started the download
- Downloading artifacts of any quantity works without issues or exceeding the memory of the gitlab host machine
Relevant logs and/or screenshots
As an example this is the memory consumption after downloading multiple artifacts summing up to about 4GB:
This is the system on idle:
Some application_json logs: 16.2.1:
{"severity":"WARN","time":"2023-08-02T07:28:06.476Z","memwd_max_strikes":5,"memwd_cur_strikes":6,"message":"rss memory limit exceeded","memwd_rss_bytes":4495462400,"memwd_max_rss_bytes":1258291200,"memwd_handler_class":"Gitlab::Memory::Watchdog::Handlers::PumaHandler","memwd_sleep_time_s":60,"pid":6267,"worker_id":"puma_1"}
{"severity":"INFO","time":"2023-08-02T07:28:06.481Z","message":"stopped","memwd_reason":"successfully handled","memwd_handler_class":"Gitlab::Memory::Watchdog::Handlers::PumaHandler","memwd_sleep_time_s":60,"pid":6267,"worker_id":"puma_1","memwd_rss_bytes":4495462400}
{"severity":"WARN","time":"2023-08-02T07:28:06.813Z","memwd_max_strikes":5,"memwd_cur_strikes":6,"message":"rss memory limit exceeded","memwd_rss_bytes":1932603392,"memwd_max_rss_bytes":1258291200,"memwd_handler_class":"Gitlab::Memory::Watchdog::Handlers::PumaHandler","memwd_sleep_time_s":60,"pid":7251,"worker_id":"puma_0"}
{"severity":"INFO","time":"2023-08-02T07:28:06.819Z","message":"stopped","memwd_reason":"successfully handled","memwd_handler_class":"Gitlab::Memory::Watchdog::Handlers::PumaHandler","memwd_sleep_time_s":60,"pid":7251,"worker_id":"puma_0","memwd_rss_bytes":1932603392}
{"severity":"DEBUG","time":"2023-08-02T07:28:08.495Z","message":"ActiveRecord connection established"}
{"severity":"INFO","time":"2023-08-02T07:28:08.608Z","message":"started","memwd_handler_class":"Gitlab::Memory::Watchdog::Handlers::PumaHandler","memwd_sleep_time_s":60,"pid":11223,"worker_id":"puma_0","memwd_rss_bytes":748306432}
{"severity":"DEBUG","time":"2023-08-02T07:28:08.666Z","message":"ActiveRecord connection established"}
{"severity":"INFO","time":"2023-08-02T07:28:08.712Z","message":"started","memwd_handler_class":"Gitlab::Memory::Watchdog::Handlers::PumaHandler","memwd_sleep_time_s":60,"pid":11238,"worker_id":"puma_1","memwd_rss_bytes":747118592}
{"severity":"WARN","time":"2023-08-02T07:59:09.390Z","memwd_max_strikes":5,"memwd_cur_strikes":6,"message":"rss memory limit exceeded","memwd_rss_bytes":4492374016,"memwd_max_rss_bytes":1258291200,"memwd_handler_class":"Gitlab::Memory::Watchdog::Handlers::PumaHandler","memwd_sleep_time_s":60,"pid":11238,"worker_id":"puma_1"}
{"severity":"INFO","time":"2023-08-02T07:59:09.395Z","message":"stopped","memwd_reason":"successfully handled","memwd_handler_class":"Gitlab::Memory::Watchdog::Handlers::PumaHandler","memwd_sleep_time_s":60,"pid":11238,"worker_id":"puma_1","memwd_rss_bytes":4492374016}
{"severity":"DEBUG","time":"2023-08-02T07:59:10.473Z","message":"ActiveRecord connection established"}
{"severity":"INFO","time":"2023-08-02T07:59:10.561Z","message":"started","memwd_handler_class":"Gitlab::Memory::Watchdog::Handlers::PumaHandler","memwd_sleep_time_s":60,"pid":15288,"worker_id":"puma_1","memwd_rss_bytes":747384832}
and even if the memory didn't exceed 12GB we get the messages rss memory limit exceeded