Pre environment enhancement for when QA runs.
The problem
The pre-environment has fewer resources than other environments. This, however, makes sense. Is a bit of a burden when QA runs, for example. This sometimes causes OOM
issues and 502
on the QA jobs.
We fixed this by bumping the pod's mem/CPU several times, as seen in 1 and 2, 3. But it's a recurring theme now; after some time, the issue gets re-opened with the modification to QA that happens by Quality.
Addressed in #19491 (closed)
Proposal
Adjust the HPA sensitivity to scale up pods for pre-components (Web, Api, WebSockets..) when needed.
Right now, the threshold is a bit high, not non-existent, and the pods are stuck at two pods maximum for web
and API,
where the QA job hits the most.
At the moment:
For API mem
:
And for Websockets mem
:
...