Skip to content

Check for spot instance interruption notice for spot instances

Sven Geislerrequested to merge
sge70/aws:sge70-spot-instance-handling into main
  • Please check this box if this contribution uses AI-generated content (including content generated by GitLab Duo features) as outlined in the GitLab DCO & CLA. As a benefit of being a GitLab Community Contributor, you can request access to GitLab Duo.

Hi @ajwalker & @josephburnett,

I try to address reports regarding spot instance termination: #35 (closed), #31 (closed), gitlab-org/gitlab-runner#38271 (closed)

Basically the change checks on every update the state of the spot instance request for related spot instances. The change does not require any change to the fleeting interface. It is entirely handled inside of the plugin.

I also included the idea from !35 (closed) to check on ConnectInfo if the instance received an interruption notice.

There are two upfront information of interested:

FYI: There are 3 ways to detect an upcoming spot instance interruption

  • on the instance you can check the meta-data for the "Spot Instance interruption notices" or the "EC2 instance rebalance recommendations" (both do require login into the instance and the request to the meta-data service)
  • you can by check the notice and the recommendation by using Amazon EventBridge (this does need a setup of an EventBridge rule by the user)
  • the "Spot Instance interruption notices" can be also checked by retrieving the state of the spot instance request (I picked this way) To reduce the number of SDK calls to the bear minimum I introduce ONE request to retrieve all related spot instance request if spot instances are present on every Update.

In a little POC the gitlab-runner didn't used anymore an instance once the "Spot Instance interruption notices" was recognized but before the instance was actually terminated.

I'm happy to discuss other options and to hear your opinion.

Add implementation for Heartbeat() introduced by gitlab-org/fleeting/fleeting!54 (merged) / gitlab-org/fleeting/fleeting#49 (closed)

Sven Geisler <sven.geisler@mercedes-benz.com>, Mercedes-AMG GmbH, Provider Information

Related to #31 (closed)

Edited by Sven Geisler

Merge request reports