Potential division-by-zero in tiff2pdf.c
Summary
Dear authors,
There exists a potential division by zero error in function t2p_readwrite_pdf_image in tools/tiif2pdf.c when compressing JPEG format.
If the value h_samp and v_samp read from input is 0, it can cause division by zero error at line 2659 and 2661
2651 TIFFGetField(input, TIFFTAG_YCBCRSUBSAMPLING, &h_samp,
2652 &v_samp);
...
2659 ri = (t2p->tiff_width + h_samp - 1) / h_samp;
2660 TIFFGetField(input, TIFFTAG_ROWSPERSTRIP, &rows);
2661 ri *= (rows + v_samp - 1) / v_samp;
Before it is called, t2p_readwrite_pdf_image_tile in tools/tiif2pdf.c does skip setting the field if it is 0, but this does not prevent the value at tag TIFFTAG_YCBCRSUBSAMPLING being 0 in line 3642.
A very similar error was previously found at the JPEG processing software Jasper.
The corresponding patch guards the hsamp and vsamp values to be between 1 to 254 to cope with the XRsiz and YRsiz range of JPEG2000 format standard while preventing the division by zero.
The possible patch would be adding similar if-guards
if (h_samp == 0 || h_samp > 255) {
TIFFError(TIFF2PDF_MODULE, ...);
return;
}
if (v_samp == 0 || v_samp > 255) {
TIFFError(TIFF2PDF_MODULE, ...);
return;
}
Version
libtiff-opengl 4.5.0
Steps to reproduce
The following error was discovered via the static analyzer, and the error trace is manually reviewed.