Jobs on Docker runner fail to run, logging "Error: No such image", docker inspect returns 'Error: No such object'

An apparent issue with of the dockerd data could result in images being inaccessible; such as the helper image. This causes jobs to fail to run.

Seen with Server Version: 1.13.1 (2017-02-08)

The following can be observed running docker commands on the runner host:

     # docker pull indicates the image is downloaded
# docker pull registry.gitlab.com/gitlab-org/gitlab-runner/gitlab-runner-helper:x86_64-943fc252
Trying to pull repository registry.gitlab.com/gitlab-org/gitlab-runner/gitlab-runner-helper ...
x86_64-943fc252: Pulling from registry.gitlab.com/gitlab-org/gitlab-runner/gitlab-runner-helper
Digest: sha256:082385b4299f264e16a04bc9e0f289a6cb3675017666a0c314e6580a3bfb2776
Status: Image is up to date for registry.gitlab.com/gitlab-org/gitlab-runner/gitlab-runner-helper:x86_64-943fc252
     # the image is queryable ..
# docker history registry.gitlab.com/gitlab-org/gitlab-runner/gitlab-runner-helper:x86_64-943fc252
IMAGE CREATED CREATED BY SIZE COMMENT
c9ad251f9d8b 2 weeks ago 66.8 MB Imported from -
     # then using the SHA reference from this output:
# docker inspect c9ad251f9d8b
[]
Error: No such object: c9ad251f9d8b

It was resolved by running:

docker system prune 
docker system prune -a

Existing content to review:

• gitlab-runner#2980 (comment 106845694)
• gitlab-runner#2980 (comment 176071406)
• gitlab-runner#2980 (comment 474850593)
• gitlab-runner#27332
• gitlab-runner-docker-cleanup#10

Ticket: 184915

Agent: Ben Prescott