Fixtures cause tmpdir to be added to `sys.path` twice

I think I saw this while working on the CPython port but forgot to follow up on it. There's a poor separation of concerns between fixtures.tempdir() and fixtures.SiteDir.site_dir(). Both add the temp dir to sys.path but only one removes it. This condition has led to test failures in the CPython project due to that test suite (rightly) detecting that sys.path is modified.