[Compressed Storage] Use smaller type of Index & StorageIndex for determining maximum size during resize.
What does this implement/fix?
Eigen::SPQR sets the StorageIndex
type always to SuiteSparse_long
.
When Eigen::Index
is set to a smaller integer type (eg int32_t
via -DEIGEN_DEFAULT_DENSE_INDEX_TYPE=int32_t
),
then the call to std::min<Index>(NumTraits<StorageIndex>::highest())
will overflow
to a negative value which then results in internal::throw_std_bad_alloc()
being called.
The proposed change uses the smaller index type to query highest()
.
Additional information
Built and ran Eigen unit tests. Additionally, this fix also works in our codebase.
Edited by Andreas Forster