Runners failing to provision

E.g. https://gitlab.com/redhat/red-hat-ci-tools/kernel/cki-internal-pipelines/cki-internal-contributors/-/jobs/8520790021 :

ERROR: Preparation failed: exit status 1
Will be retried in 3s ...
ERROR: Preparation failed: exit status 1
Will be retried in 3s ...
ERROR: Preparation failed: exit status 1
Will be retried in 3s ...
ERROR: Job failed (system failure): exit status 1

https://us-east-1.console.aws.amazon.com/cloudtrailv2/home?region=us-east-1#/events?EventName=CreateFleet

At first it seemed that we were hitting "UnfulfillableCapacity" in our spot instance requests, which could be happening in decurrence of "AWS re:invent".

Now we think it might have been caused by a Fedora 39 AMI getting deleted without us preparing for it.

Upstream ticket to make it possible to copy Fedora AMIs: https://pagure.io/fedora-infrastructure/issue/12320

Edited by Michael Hofmann
To upload designs, you'll need to enable LFS and have an admin enable hashed storage. More information