fix for denial of service due to mishandled out of memory

Hi, I find a potential remote denial of service issue caused by the mishandled out of memory situation.

The problem happens in the JBIGDecode function in libtiff/tif_jbig.c file.

At line 78, the jbg_dec_in function aborts when out of memory.

However, the function itself does not want the program stops according to the function semantic.

To reproduce the case, you need to compile the libtiff with:

CFLAGS="-fsanitize=address,leak,undefined"

then run:

./tiff2ps input

The trace reported by ASAN:

==56365==AddressSanitizer CHECK failed: ../../../../src/libsanitizer/sanitizer_common/sanitizer_allocator.cc:147 "((0)) != (0)" (0x0, 0x0)
    #0 0x7f2e33903631  (/usr/lib/x86_64-linux-gnu/libasan.so.2+0xa0631)
    #1 0x7f2e339085e3 in __sanitizer::CheckFailed(char const*, int, char const*, unsigned long long, unsigned long long) (/usr/lib/x86_64-linux-gnu/libasan.so.2+0xa55e3)
    #2 0x7f2e33880425  (/usr/lib/x86_64-linux-gnu/libasan.so.2+0x1d425)
    #3 0x7f2e33906865  (/usr/lib/x86_64-linux-gnu/libasan.so.2+0xa3865)
    #4 0x7f2e33885b4d  (/usr/lib/x86_64-linux-gnu/libasan.so.2+0x22b4d)
    #5 0x7f2e338fb5d2 in malloc (/usr/lib/x86_64-linux-gnu/libasan.so.2+0x985d2)
    #6 0x7f2e334342fd  (/usr/lib/x86_64-linux-gnu/libjbig.so.0+0x12fd)
    #7 0x7f2e33439a17 in jbg_dec_in (/usr/lib/x86_64-linux-gnu/libjbig.so.0+0x6a17)
    #8 0x6be3cc in JBIGDecode /mnt/data/playground/tiff-4.0.10-a/libtiff/tif_jbig.c:78
    #9 0x567502 in TIFFReadEncodedStrip /mnt/data/playground/tiff-4.0.10-a/libtiff/tif_read.c:544
    #10 0x434f07 in PSDataBW /mnt/data/playground/tiff-4.0.10-a/tools/tiff2ps.c:2662
    #11 0x437c02 in PSpage /mnt/data/playground/tiff-4.0.10-a/tools/tiff2ps.c:2392
    #12 0x43c256 in TIFF2PS /mnt/data/playground/tiff-4.0.10-a/tools/tiff2ps.c:1610
    #13 0x40b318 in main /mnt/data/playground/tiff-4.0.10-a/tools/tiff2ps.c:477
    #14 0x7f2e3290d82f in __libc_start_main (/lib/x86_64-linux-gnu/libc.so.6+0x2082f)
    #15 0x40e7f8 in _start (/mnt/data/playground/tiff-4.0.10-a/tools/tiff2ps+0x40e7f8)

The patch might be alternative, my patch adds a signal handler during the decoding. It can also check the required buffer size before starting decoding.

The input is attached here: input