make sure we only remove the xd after all XferDesQueue::update_pre_bytes_total calls are received