Occasional macos job failures timing out on "Dialing nesting daemon"
<!--IssueSummary start--> <details> <summary> Everyone can contribute. [Help move this issue forward](https://handbook.gitlab.com/handbook/marketing/developer-relations/contributor-success/community-contributors-workflows/#contributor-links) while earning points, leveling up and collecting rewards. </summary> - [Close this issue](https://contributors.gitlab.com/manage-issue?action=close&projectId=278964&issueIid=524383) </details> <!--IssueSummary end--> <!--- Please read this! Before opening a new issue, make sure to search for keywords in the issues filtered by the "regression" or "type::bug" label: - https://gitlab.com/gitlab-org/gitlab/issues?label_name%5B%5D=regression - https://gitlab.com/gitlab-org/gitlab/issues?label_name%5B%5D=type::bug and verify the issue you're about to submit isn't a duplicate. ---> In the [saas-macos-medium-m1-basic-tests](https://gitlab.com/gitlab-org/ci-cd/tests/saas-runners-tests/macos-platform/saas-macos-medium-m1-basic-tests) project I have a scheduled pipeline running every other hour. We are getting an occasional job failures, like so: - https://gitlab.com/gitlab-org/ci-cd/tests/saas-runners-tests/macos-platform/saas-macos-medium-m1-basic-tests/-/jobs/8954071551 - https://gitlab.com/gitlab-org/ci-cd/tests/saas-runners-tests/macos-platform/saas-macos-medium-m1-basic-tests/-/jobs/8952695364 - https://gitlab.com/gitlab-org/ci-cd/tests/saas-runners-tests/macos-platform/saas-macos-medium-m1-basic-tests/-/jobs/8948890215 - https://gitlab.com/gitlab-org/ci-cd/tests/saas-runners-tests/macos-platform/saas-macos-medium-m1-basic-tests/-/jobs/8941796715 __This is impacting only 1% of all jobs currently__ All of these failures are in the form of: ```shell Running with gitlab-runner 17.7.0~pre.103.g896916a8 (896916a8) on blue-2.saas-macos-medium-m1.runners-manager.gitlab.com/default tCo7q1eW, system ID: s_f10a77c3f6e9 Resolving secrets Preparing the "instance" executor 01:00:00 Preparing instance... Dialing instance i-06ba8edc53bb73918... Instance i-06ba8edc53bb73918 connected Enforcing VM Isolation Creating nesting VM tunnel Creating nesting VM macos-14-xcode-15 Dialing nesting daemon ERROR: Job failed: execution took longer than 1h0m0s seconds ``` These start to build up on the macos instances that have been active the most and for the longest time period. This is a host degradation issue. Most likely we can detect this repetitive failure and delete the host as suggested in https://gitlab.com/gitlab-org/fleeting/taskscaler/-/issues/9+ <!-- and uncomment below if you have /label privileges --> <!-- or follow up with an issue comment of `@gitlab-bot label ~"reproduced on GitLab.com"` if you do not --> <!-- If you don't have /label privileges, follow up with an issue comment of `@gitlab-bot label ~"type::bug"` -->
issue