Fix integer overflow in _TIFFCheckMalloc() and other implementation-defined behaviour (CVE-2019-14973)

Even Rouault requested to merge rouault/libtiff:fix_integer_overflow into master

_TIFFCheckMalloc()/_TIFFCheckRealloc() used a unsafe way to detect overflow in the multiplication of nmemb and elem_size (which are of type tmsize_t, thus signed), which was especially easily triggered on 32-bit builds (with recent enough compilers that assume that signed multiplication cannot overflow, since this is undefined behaviour by the C standard). The original issue which lead to this fix was trigged from tif_fax3.c

There were also unsafe (implementation defied), and broken in practice on 64bit builds, ways of checking that a uint64 fits of a (signed) tmsize_t by doing (uint64)(tmsize_t)uint64_var != uint64_var comparisons. Those have no known at that time exploits, but are better to fix in a more bullet-proof way. Or similarly use of (int64)uint64_var <= 0.

Edited by Even Rouault

Merge request reports