core: simplify code, add asserts, remove undefined behavior around int64x64-128.

1. Fix an overflow bug in UmulByInvert() where result can differ by 2^64 from intended.
2. Add asserts to detect overflows in Umul() and UmulByInvert().

This is related to !1010 (closed).

1. Add asserts to detect overflows in Mul(), Umul() to ensure the number's product can be represented by int64x64_t
2. Add an assert in UmulByInvert() to check b came from Invert()
3. Fix undefined behavior when negating the smallest signed integer in output_sign() and MulByInvert()
4. Improve documentation in the code