Add `fallback region` setting for docker-machine executor

Overview

For the google driver or any other cloud provider driver, you can specify a zone/region. In some cases, this zone might start misbehaving which renders the CI system to be unusable because it can't spawn new machines in the faulty zone. This would require manual intervention from the Runner admins to change to a zone that is healthy.

Have GitLab Runner look at the error rates generated on machine creatin/deletion and implement a circuit breaker that if errors are X amount in X minutes fallback to another region that is specified inside of the config.toml

Things to keep in mind

  • How will GitLab Runner know which field to set for the docker-machine to change since for each driver it is different? For example the google driver uses --google-zone whilst aws uses --amazonec2-zone.
  • When should we go back to the original/preferred zone?
  • How will we indicate to the user what we change the zone, is a log message enough?
  • We're not trying to add regional failover to the existing setup. This is out of scope.
Edited by Alan Richards