Track number of failure on docker-machine creation
Overview
In gitlab-com/gl-infra/production#4649 (closed) we saw that we two gitlab-runner
instances where it failing to create a ton of machines. We only saw this when diving deep into the logs which took time and wasn't visabile immedtly.
Proposal
Create a counter metric to increment by 1 for every machine we failed to create. Like this we can have this inside of Grafana and track the number of machine failures.
If possible we should also consider creating a counter for each operation
that we do with docker-machine
such as create
, delete
, provision
.