Use std::unordered_map and std::unordered_set instead of std::map / std::set
std::map and std::set allow ordered iteration, which is pointless with pointers. Instead, std::unordered_map / std::unordered_set have constant complexity for most operations instead of logarithmic with std::map / std::set.