Remove build dirs in CI scripts

In CI, this won't make a difference, but I've started using
gitlabci-local to run CI builds locally in ephemeral containers and this
makes that process slightly smoother.