Start cache cleaner only in main executable (not test)

Possible fix for #1872 (closed)

Alternative to !1437 (closed)

I think !1437 (closed) is right to call out the status quo of how we manage config in memory as "brittle". The approach in this MR does nothing to fix the brittleness, it just fixes the one thing that is currently broken.