GitLab Runner Autoscaler: Enable horizontal scaling based on resource usage

Description

Currently, the GitLab Runner Autoscaler scales instances based on the number of jobs. The challenge is that resource requirements vary between jobs, which can lead to instances being overutilized or underutilized even with the same number of jobs assigned to them. Setting resource limits per job/container does not fully help here as well.

Proposal

It would be great if the autoscaler could monitor resource usage in existing instances and scale out more when consumption exceeds a configured limit. This approach is similar to how Kubernetes scales nodes based on resource consumption in some cloud providers. For example: Scale cluster compute with Karpenter and Cluster Autoscaler.

Links to related issues and merge requests / references

A 1850 seater GitLab Premium customer is interested in this feature. Their use case is to build a standby, Docker-based, autoscaled Runner infrastructure that can serve as a failover if their main Kubernetes Runner infrastructure becomes unavailable.

ZD for internal use.