Container Registry being OOM killed often
Recent issues have surfaced due to the Container Registry being OOM killed due to node pressure. This has led to high evictions. We need to take action against this now. Currently we are never reaching our memory limits, and instead our node falls under pressure first. This is currently set to 4GB, but as seen below we aren't even reaching this before the Pod or in some cases, many Pods are Evicted at near the same time.
A fix we can apply sooner than existing issues would be to adjust memory limits to a much lower value than 4GB which will stagger the Pod Evictions due to the Pods ramping up their memory usage differently over the course of time. Let's attempt to do this now, we can re-evaluate the following issues after some time has passed: