Don't keep MetaSource around in Source

We can easily rebiuld MetaSource from any source so there is no
reason to keep them around.

This will slightly improve memory usage.