Build dependency cache eagerly

We can update the cache as we build the elements, we don't need
to wait until we have built the whole graph