Skip to content

Draft: Implement docker/wait.RemoveWait and wait for containers to be removed during cleanup

Arran Walker requested to merge 27022-failed-to-cleanup-volumes into main

What does this MR do?

Implements wait.RemoveWait and waits for containers to be removed during cleanup.

Why was this MR needed?

Containers are removed at the end of the build in Cleanup(). However, removeContainer only issues the command to remove, it doesn't block until the container has been fully removed. It doesn't immediately return either, but, it also doesn't block until absolutely complete. Additional cleanup commands executed afterwards expect the container to have been removed. This blocks as intended, until timeout.

What's the best way to test this MR?

There's some tests for ensuring WaitRemove works as expected.

For manual testing, running any manual job and waiting for containers to be successfully removed should be enough.

In #27022 (closed), we have a scenario where volumes cannot be cleaned correctly because they're "still in use". I believe this is due to the container still not being removed, which is where this MR should help.

Unfortunately, I'm unable to now reproduce this, and therefore test this solves the problem.

I'd hoped creating the following job would work:

my-job:
  image: "busybox:latest"
  script:
    # create a bunch of nested directories and a file
    # this should take around about ~2 minutes to create, and a similar time to delete (which does does for us, during cleanup)
    - start=$(date +%s)
    - for i in `seq 120000`; do mkdir -p nuke/$i/a/b/c/d/e/f/g/h/i/; dd if=/dev/zero of=nuke/$i/a/b/c/d/e/f/g/h/i/file bs=2048 count=1 2> /dev/null; done
    - echo "$(($(date +%s) - ${start}))"

However, this actually returns a different error stating the docker daemon cannot be accessed. It appears volumes that take a very long time to be removed completely lock up the daemon for any further volume operations.

What are the relevant issue numbers?

Closes #27022 (closed)

Edited by Arran Walker

Merge request reports