Remove recently added sparse assert in SparseMapBase.

Turns out we have existing use-cases where we use the map to populate the sparse matrix, so the map may not be valid on construction here.