We now delay refreshing submodules until we have a clone, that is after we have fetched the main fetcher. Fixes #537