fix numtrunc at tiff_dirread.c

Hi! We've been fuzzing opencv with sydr-fuzz security predicates and we found numeric truncation error in libtiff as 3rd party of opencv in tif_dirread.c:4386.

In function TIFFReadDirectoryCheckOrder on line 4386 variable m has type uint16, structure TIFFDirEntry has field uint16 tdir_tag. But on the right side of operator there is an integer type value (due to the integer promotion), so the numeric truncation may occur. Our tool has found input when o->tdir_tag is equal to 65535 (uint16_max), so then value o->tdir_tag+1 truncates to zero in variable m. Then m is used in if operator on line 4380, and it is obvious, that o->tdir_tag<m can't be true after truncation and the break from the cycle will not occur even with not ascending order of tags. So we suggest to change the type int16_t of the variable m to type uint32.

Environment

• OS: ubuntu 20.04
• commit: a42b35598e2543d6dab3ac73be92fb77742e0747 (opencv)

How to reproduce this error

1. Build docker container:

   sudo docker build -t oss-sydr-fuzz-opencv .
   

2. Run docker container:

   sudo docker run --privileged --network host -v /etc/localtime:/etc/localtime:ro --rm -it -v $PWD:/fuzz oss-sydr-fuzz-opencv /bin/bash
   

3. Run on the following input:

    /out/imread_fuzzer sydr_ce171c30cd94046b3233028c74c7fce1a956c7e1_num_trunc_0_unsigned.txt
   

4. Output:

   /opencv/3rdparty/libtiff/tif_dirread.c:4386:5: runtime error: implicit conversion from type 'int' of value 65536 (32-bit, signed) to type 'uint16' (aka 'unsigned short') changed the value to 0 (16-bit, unsigned)
   SUMMARY: UndefinedBehaviorSanitizer: undefined-behavior /opencv/3rdparty/libtiff/tif_dirread.c:4386:5