Update mtimes of objects for requested references.

This also remove references when some objects are missing. This is in
preparation for the move from reference to object garbage collection.