CI: docker-rpi-builder, C1 instances are all in Stopped state
Summary
Twice in two weeks, we've found that docker-machine
on our docker-rpi=builder
tagged CI jobs are "stuck" in a complete Stopped
state.
- 2020-06-29 https://dev.gitlab.org/gitlab/omnibus-gitlab/-/pipelines/159839/failures
- 2020-07-07 https://dev.gitlab.org/gitlab/omnibus-gitlab/-/pipelines/160746/failures
When performing our documented method for removing them, docker-machine
got HTTP 400
responses.
Details
docker-machine ls | grep -v 'Running'
NAME ACTIVE DRIVER STATE URL SWARM DOCKER ERRORS
runner-8d568bd2-omnibus-builder-1593548570-22cc68e5-C1 - scaleway(C1) Stopped Unknown
runner-8d568bd2-omnibus-builder-1593548574-0f29ea6a-C1 - scaleway(C1) Stopped Unknown
runner-8d568bd2-omnibus-builder-1593548574-78c09cd4-C1 - scaleway(C1) Stopped Unknown
runner-8d568bd2-omnibus-builder-1593548578-4e72e2e8-C1 - scaleway(C1) Stopped Unknown
runner-8d568bd2-omnibus-builder-1593548578-41a2ae13-C1 - scaleway(C1) Stopped Unknown
runner-8d568bd2-omnibus-builder-1593549074-d2fde370-C1 - scaleway(C1) Stopped Unknown
runner-8d568bd2-omnibus-builder-1593550666-222025c3-C1 - scaleway(C1) Stopped Unknown
runner-8d568bd2-omnibus-builder-1593954451-7f2cf347-C1 - scaleway(C1) Stopped Unknown
runner-8d568bd2-omnibus-builder-1593954451-5453dc6f-C1 - scaleway(C1) Stopped Unknown
runner-8d568bd2-omnibus-builder-1593955512-ce682f2e-C1 - scaleway(C1) Stopped Unknown
runner-ddcyweuy-scaleway-dev-gitlab-1593521911-4e397e96-C1 - scaleway(C1) Stopped Unknown
runner-ddcyweuy-scaleway-dev-gitlab-1593548571-f93630b9-C1 - scaleway(C1) Stopped Unknown
output from removing Stopped instances
Error removing host "runner-8d568bd2-omnibus-builder-1593548570-22cc68e5-C1": StatusCode: 400, Type: invalid_request_error, APIMessage: server should be running Successfully removed runner-8d568bd2-omnibus-builder-1593548570-22cc68e5-C1 Error removing host "runner-8d568bd2-omnibus-builder-1593548574-0f29ea6a-C1": StatusCode: 400, Type: invalid_request_error, APIMessage: server should be running Successfully removed runner-8d568bd2-omnibus-builder-1593548574-0f29ea6a-C1 Error removing host "runner-8d568bd2-omnibus-builder-1593548574-78c09cd4-C1": StatusCode: 400, Type: invalid_request_error, APIMessage: server should be running Successfully removed runner-8d568bd2-omnibus-builder-1593548574-78c09cd4-C1 Error removing host "runner-8d568bd2-omnibus-builder-1593548578-4e72e2e8-C1": StatusCode: 400, Type: invalid_request_error, APIMessage: server should be running Successfully removed runner-8d568bd2-omnibus-builder-1593548578-4e72e2e8-C1 Error removing host "runner-8d568bd2-omnibus-builder-1593548578-41a2ae13-C1": StatusCode: 400, Type: invalid_request_error, APIMessage: server should be running Successfully removed runner-8d568bd2-omnibus-builder-1593548578-41a2ae13-C1 Error removing host "runner-8d568bd2-omnibus-builder-1593549074-d2fde370-C1": StatusCode: 400, Type: invalid_request_error, APIMessage: server should be running Successfully removed runner-8d568bd2-omnibus-builder-1593549074-d2fde370-C1 Error removing host "runner-8d568bd2-omnibus-builder-1593550666-222025c3-C1": StatusCode: 400, Type: invalid_request_error, APIMessage: server should be running Successfully removed runner-8d568bd2-omnibus-builder-1593550666-222025c3-C1 Error removing host "runner-8d568bd2-omnibus-builder-1593954451-7f2cf347-C1": StatusCode: 400, Type: invalid_request_error, APIMessage: server should be running Successfully removed runner-8d568bd2-omnibus-builder-1593954451-7f2cf347-C1 Error removing host "runner-8d568bd2-omnibus-builder-1593954451-5453dc6f-C1": StatusCode: 400, Type: invalid_request_error, APIMessage: server should be running Successfully removed runner-8d568bd2-omnibus-builder-1593954451-5453dc6f-C1 Error removing host "runner-8d568bd2-omnibus-builder-1593955512-ce682f2e-C1": StatusCode: 400, Type: invalid_request_error, APIMessage: server should be running Successfully removed runner-8d568bd2-omnibus-builder-1593955512-ce682f2e-C1 Error removing host "runner-ddcyweuy-scaleway-dev-gitlab-1593521911-4e397e96-C1": StatusCode: 400, Type: invalid_request_error, APIMessage: server should be running Successfully removed runner-ddcyweuy-scaleway-dev-gitlab-1593521911-4e397e96-C1 Error removing host "runner-ddcyweuy-scaleway-dev-gitlab-1593548571-f93630b9-C1": StatusCode: 400, Type: invalid_request_error, APIMessage: server should be running Successfully removed runner-ddcyweuy-scaleway-dev-gitlab-1593548571-f93630b9-C1
Edited by Jason Plum