RTMPT dissector inf loop - 100% cpu - denial of service
Summary
It is possible to reach an infinite loop in the RTMPT dissector by generating a specifically crafted RTMP packet due to int underflow vulnerability in the AMF parsing part of the dissector. The packet will consume 100% core cpu, which eventually lead to a denial of service via packet injection or crafted capture file.
I believe that the bug was introduced to the code following this commit - #5351 (comment 400649448)
If that's true, it means that the bug exists for approximately 11 years over many tshark releases.
Technical details
Real-Time Messaging Protocol (RTMP) is a communication protocol for streaming audio, video, and data over the Internet which was mainly used by Flash Player. RTMP is a binary protocol over tcp port 1935.
The protocol enables to encode Action Message Format (AMF) data and the parsing of AMF object is mainly done by the rtmpt_get_amf_param or rtmpt_get_amf_txid functions. Before parsing AMF param, it is needed to understand its size. Therefore, rtmpt_get_amf_length is called to get the item param size. However, it is possible to reach an infinite loop condition within rtmpt_get_amf_length by carefully crafting AMF message as follows:
The function has a main while loop with the following conditions: while (rv == 0 || depth > 0) so we need to make sure rv == 0. In the first round all variable are initialised to 0.
The first meaningful line reads the object type:
iObjType = tvb_get_guint8(tvb, offset+rv);
Since we control the data, we can encode a speicifc object type that has a dynamic size. For example - AMF0_STRING (+3) or AMF0_XML (+5). These types allow to read the item length from the buffer using tvb_get_ntohl. However, itemlen is defined as a signed int gint, which means we can cause an int underflow by providing a size such as 0xfffffffd (-3) 0xfffffffb (-5). Then, since itemlen will remain zero, the loop will repeat itself because rv is affected by the itemlen:
rv += itemlen; and so rv will remain zero forever and the loop will continue forerver.
Relevant functions:
rtmpt_get_amf_txid --> rtmpt_get_amf_length
Steps to reproduce
Open the provided pcaps with tshark/wireshark poc
What is the current bug behavior?
Wireshark/tshark will be stuck in an endless loop due to an int underflow bug.
What is the expected correct behavior?
Wireshark/tshark shouldn't be stuck in an endless loop.
Sample capture file
Attached poc
Relevant logs and/or screenshots
Build information
Version 3.6.0 (v3.6.0-0-g3a34e44d02c9)