Fix inverse nullptr/asan errors.

For empty or single-column matrices, the current PartialPivLU currently dereferences a nullptr or accesses memory out-of-bounds. Here we adjust the checks to avoid this.