TIFFRasterScanlineSize64 produce too-big size and could cause OOM
# Summary
`TIFFRasterScanlineSize64` could produce a very huge size with a crafted tiff input.
Perform allocation on the size returned by `TIFFRasterScanlineSize64` could cause OOM.
Remote attackers could utilize this bug cause deny-of-services.
# Version
commit: 4d0329a4539550f2396772b8c4c60c5fecdda7db Sep 15
# POC program
```
// poc.cc
#include <tiffio.hxx>
#include <stdlib.h>
#include <string.h>
#include <stdint.h>
#include <vector>
#include <fstream>
#include <iostream>
#include <sstream>
extern "C" int LLVMFuzzerTestOneInput(const uint8_t* data, size_t size) {
if(size<=0) return 0;
// open input tiff in memory
std::istringstream s(std::string(data, data + size));
TIFF *in_tif = TIFFStreamOpen("MemTIFF", &s);
if (!in_tif) {
return 0;
}
tdir_t directoryCount = TIFFNumberOfDirectories(in_tif);
for (tdir_t i = 0; i < directoryCount; i++) {
if (TIFFSetDirectory(in_tif, i)) {
tmsize_t scanlineSize = TIFFRasterScanlineSize(in_tif);
if (scanlineSize > 0) {
uint8_t* scanline = (uint8_t*)_TIFFmalloc(scanlineSize);
tmsize_t bytesRead = TIFFReadScanline(in_tif, scanline, TIFFCurrentRow(in_tif), 0);
if (bytesRead >= 0) {
// Process the scanline data
// ...
}
_TIFFfree(scanline);
}
}
}
TIFFClose(in_tif);
return 0;
}
```
# Details about the bug
As referred from the document of the API [TIFFRasterScanlineSize](http://www.simplesystems.org/libtiff/functions/TIFFsize.html?highlight=tiffrasterscanlinesize#c.TIFFRasterScanlineSize):
> TIFFRasterScanlineSize() returns the size in bytes of a complete decoded and packed raster scanline.
The API `TIFFRasterScanlineSize` returns a size of bytes.
However, the size returned by `TIFFRasterScanlineSize` can be controlled by user input and the size is not checked.
`TIFFRasterScanlineSize` calls `TIFFRasterScanlineSize64()` and `TIFFRasterScanlineSize64` calculate the size by the user controllable fields: ` td->td_bitspersample` and `td->td_imagewidth`.
In the [triger_input_41](https://github.com/PromptFuzz/crash_inputs/raw/main/libtiff/oom1/triger_input_41), their offsets are 0xC2-0xC3 and 0xAA-0xAb respectively.
```
uint64_t TIFFRasterScanlineSize64(TIFF *tif)
{
static const char module[] = "TIFFRasterScanlineSize64";
TIFFDirectory *td = &tif->tif_dir;
uint64_t scanline;
scanline =
_TIFFMultiply64(tif, td->td_bitspersample, td->td_imagewidth, module);
if (td->td_planarconfig == PLANARCONFIG_CONTIG)
{
scanline =
_TIFFMultiply64(tif, scanline, td->td_samplesperpixel, module);
return (TIFFhowmany8_64(scanline));
}
else
return (_TIFFMultiply64(tif, TIFFhowmany8_64(scanline),
td->td_samplesperpixel, module));
}
```
If the user controls it returns a huge size, the successive allocations on this size could cause OOMs.
> libtiff/tools/tiffcp.c also contains such a usage. This usage is not a rare scenario.
# Steps to reproduce
1. Download the poc input: [triger_input_41](https://github.com/PromptFuzz/crash_inputs/raw/main/libtiff/oom1/triger_input_41)
(How one can reproduce the issue - this is very important)
2. Built libtiff with ASAN
3. Compile the Poc program:
```
clang++ -fsanitize=fuzzer,address -g -O0 -I/libtiff/include poc.cc -o poc.out libtiff.a -lz -ljpeg -llzma -ljbig
```
4. Trigger the bug:
```
./poc.out triger_input_41
```
# Platform
Ubuntu 22.04 Clang 15.0.0
issue