Docker-machine Preparation Failed

Summary

Successfully preparing the docker-machine, but failing to prepare the wrapped executor (docker) appears to happen frequently.

The Prepare retry seems to always succeed. However, during the retry we increment the docker-machine's UsedCount.

https://gitlab.com/gitlab-org/gitlab/-/jobs/630813484:

ERROR: Preparation failed: adding cache volume: set volume permissions: create permission container for volume "runner-ae3bfce2-project-278964-concurrent-0-cache-3c3f060a0374fc8bc39395164f415a70": Error response from daemon: mkdir /var/lib/docker/overlay2/0e36925103dac8982c68af023828dbfdc07084374e45d1efa434cea6646b2269-init: read-only file system (linux_set.go:90:0s)

https://gitlab.com/gitlab-org/gitlab/-/jobs/630244445:

ERROR: Preparation failed: adding cache volume: set volume permissions: running permission container "05cc1b9813a2e5190b1ceaae8a47ff3ac7bc28a0bdbdf731f573ab36b39a726d" for volume "runner-9538b0ab-project-278964-concurrent-0-cache-c33bcaa1fd2c77edfc3893b41966cea8": starting permission container: Error response from daemon: error setting label on mount source '/var/lib/docker/volumes/runner-9538b0ab-project-278964-concurrent-0-cache-c33bcaa1fd2c77edfc3893b41966cea8/_data': bad message (linux_set.go:100:9s)

Steps to reproduce

I cannot reproduce this locally. My guess is that maybe it only occurs when using docker-machine + GCP.

Maybe the disk isn't yet ready? A retry always works.

Proposal

• placeholder for implementation proposal}