Invalidate caches if the cache refers to a pybuild file at a different path

Fixes an issue brought up by @jan.thomas on matrix. The cache is validated only based on the files which are listed in the cache file, but if the repository being used has changed without removing the old repository which produced the cache file, then the cache will be considered valid despite the new file being different.