Use taskscaler and nesting slots

What does this MR do?

This merge request uses the slot feature of nesting and taskscaler in order to lazily cleanup VMs and make sure the limit of 2 per instance is not exceeded.

Why was this MR needed?

If runner restarts it will lose track of VMs on the instance. It could proactively cleanup on restart, but this method is easier and more robust.

What's the best way to test this MR?

Start a build on a Mac runner. Disconnect the network and terminate the runner so it doesn't have a chance to cleanup. Verify the orphan VM is still running on the Mac (nesting list). Restart the runner and start another build. Verify the orphan VM is removed (stomped) before the new VM is created.

Here is my results from this test:

ec2-user@ip-10-0-156-130 ~ % tart list
Source  Name
local   macos-12-xcode-13
local   macos-12-xcode-14
local   nesting-nk8ifa3f
ec2-user@ip-10-0-156-130 ~ % nesting list
nesting-nk8ifa3f  192.168.64.5

Restarting the runner, I see this warning (I need to fix the message):

2023-01-19T09:39:46.409-0800 [WARN]  The following keys are present on instance %q but are not assigned slots in taskscaler: %v: %v: instance=i-0dcca5053e747a743 i-0dcca5053e747a743=["nesting-nk8ifa3f"]

On the next build the VM is stomped:

ec2-user@ip-10-0-156-130 ~ % tart list
Source  Name
local   macos-12-xcode-13
local   macos-12-xcode-14
local   nesting-b4who5g6
ec2-user@ip-10-0-156-130 ~ % nesting list
nesting-b4who5g6  192.168.64.6

And I see this message in the logs:

Created vm nesting-b4who5g6 192.168.64.6            job=3631431528 project=39258790 runner=LMPaaTzF
Stomped vm:  nesting-nk8ifa3f                       job=3631431528 project=39258790 runner=LMPaaTzF

What are the relevant issue numbers?

gitlab-org/ci-cd/shared-runners/images/macstadium/orka#109 (moved)