An out-of-order packet incorrectly detected as retransmission breaks desegmentation of TCP stream

Georg Müller said:

Build Information: TShark (Wireshark) 3.1.1 (v3.1.1rc0-126-g2a32808e)

Copyright 1998-2019 Gerald Combs <gerald@‍wireshark.org> and contributors. License GPLv2+: GNU GPL version 2 or later <https://www.gnu.org/licenses/gpl-2.0.html> This is free software; see the source for copying conditions. There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.

Compiled (64-bit) with libpcap, with POSIX capabilities (Linux), with libnl 3, with GLib 2.60.6, with zlib 1.2.11, without SMI, with c-ares 1.15.0, without Lua, with GnuTLS 3.6.8 and PKCS #‍11 support, with Gcrypt 1.8.4, with MIT Kerberos, with MaxMind DB resolver, without nghttp2, without brotli, without LZ4, without Snappy, with libxml2 2.9.9.

Running on Linux 5.1.19-300.fc30.x86_64, with Intel(R) Core(TM) i7-4600U CPU @ 2.10GHz (with SSE4.2), with 15943 MB of physical memory, with locale de_DE.UTF-8, with libpcap version 1.9.0-PRE-GIT (with TPACKET_V3), with GnuTLS 3.6.8, with Gcrypt 1.8.4, with zlib 1.2.11, binary plugins supported (0 loaded).

Built using gcc 9.1.1 20190503 (Red Hat 9.1.1-1).

I have a trace with an out-of-order packet which arrives 300ms late. Due to the late arrival but a low initial RTT (170ms), the packet is marked es retransmission in tcp_analyze_sequence_number().

In desegment_tcp(), due to the TCP_A_RETRANSMISSION flag, the data is not considered as a valid fragment.

Due to this, the segment never gets reassembled. When I disable the time check in tcp_analyze_sequence_number() or the retransmission check in desegment_tcp(), every thing works fine

Would it be possible (if desegmentation is enabled) to check if the packet fills a gap in the reassembly and then NOT flag it as retransmission, but as out-of-order? Or don't take the retransmission path in desegment_tcp()?

added libwireshark oslinux versiondev labels

Peter Wu said:

Could you attach a packet capture that reproduces this issue?

Would the "Reassemble out-of-order segments" option help? https://www.wireshark.org/docs/wsug_html_chunked/ChAdvReassemblySection.html#‍ChAdvReassemblyTcp

Georg Müller said:

Created attachment 17284 script to generate a pcap which shows the problem

Georg Müller said:

Created attachment 17285 Trace with all packets in order

Just as a reference

Georg Müller said:

Created attachment 17286 trace with two packets swapped, long initial RTT

This trace shows correct behavior.

Since the initial RTT is longer (generated using 0.3s in line 24 of the script), packet 12 is detected as out-of order.

All the other PDUs (in frame 14, 20, 26) are shown correctly.

Georg Müller said:

Created attachment 17287 trace with two packets swapped, short initial RTT, showing the problem

This trace shows incorrect behavior.

Since the initial RTT is shorter (generated using 0.1s in line 24 of the script), packet 12 is detected as retransmission.

No more PDUs get reassembled after this problem.

Georg Müller said:

(In reply to Peter Wu from comment #‍1)
> Would the "Reassemble out-of-order segments" option help?
> https://www.wireshark.org/docs/wsug_html_chunked/ChAdvReassemblySection.
> html#‍ChAdvReassemblyTcp

This option is enabled, but due to the classification as retransmission, it does not work.

Peter Wu said:

Created attachment 17386
TLS (HTTP/2 capture) with lost and retransmitted packet

Are you sure that the packets were merely out-of-order as opposed to retransmitted due to packet loss?

I ran into a real-world situation where a packet was likely lost, but quickly retransmitted. However, this somehow prevented dissection of the data due to Wireshark marking it as retransmitted. Part of the attached trace:

  4 0.061210176 A→B                 [PSH, ACK] Seq=1    Ack=1    Len=517
  5 0.150476948 B→A                      [ACK] Seq=1    Ack=518  Len=0
  6 0.151647581 B→A [TCP Previous segment not captured]
                                    [PSH, ACK] Seq=1389 Ack=518  Len=1761
  7 0.151660609 A→B [TCP Dup ACK 3#1]    [ACK] Seq=518  Ack=1    Len=0 SLE=1389 SRE=3150
  8 0.247150535 B→A [TCP Retransmission] [ACK] Seq=1    Ack=518  Len=1388
  9 0.247170838 B→A [TCP Retransmission] [ACK] Seq=1    Ack=518  Len=1388
 10 0.247196611 A→B                      [ACK] Seq=518  Ack=3150 Len=0
 11 0.247205014 A→B [TCP Dup ACK 10#1]   [ACK] Seq=518  Ack=3150 Len=0 SLE=1    SRE=1389
 12 0.251069065 A→B                 [PSH, ACK] Seq=518  Ack=3150 Len=64

Comments:

 4. Client sends data (TLS Client Hello)
 5. Note that Seq=1, Len=0. This is the server response (TCP ACK).
 6. Note that Seq=1389. That means that packet with Seq=1, Len=1388 is missing.
 7. Client detects missing packet and sends a SACK (Ack=1, but SLE=1389).
 8. Server detects missing data and retransmits it.
 9. (Retransmission of the same data from 8)
10. Client confirms receipt of data (1389-1761 from 6, 1-1388 from 8).
11. Client sends another SACK (no idea why as SRE <= Ack)
12. Clients continues with more data.

I'll try to find fix reasssembly when the "Reassemble out-of-order" option is enabled. The packet will likely still be flagged as "Retransmitted" however.

Georg Müller said:

It was no retransmission. But I am fine with it being flagged as retransmission, as long as the desegment succeeds.

If the original packet is not in the trace, but the retransmission is, then the retransmission should be used to fill the gap in the desegment buffer.

I have looked into the code. If there were a function to ask the desegment buffer if bytes from the current packet are missing, then an add should be fine.

Peter Wu said:

I was able to reassemble stuff by skipping the return here, but more work is necessary to avoid calling dissectors multiple times for the same segment.

--- a/epan/dissectors/packet-tcp.c
+++ b/epan/dissectors/packet-tcp.c
@@ -3176,7 +3176,7 @@ again:
             proto_tree_add_bytes_format(tcp_tree, hf_tcp_segment_data, tvb, offset,
                 nbytes, NULL, "%sTCP segment data (%u byte%s)", str, nbytes,
                 plurality(nbytes, "", "s"));
-            return;
+            if (!g_getenv("HACK")) return;
         }
         /* Else, find the most previous PDU starting before this sequence number */
         if (!msp) {

Georg Müller said:

Created attachment 17387 File to generate a pcap with a duplicate packet (retransmission)

Georg Müller said:

Created attachment 17388
Trace with a duplicate packet

Tests with a real retransmission of your hack-patch:

Without your patch, this trace gives the following output:

   11   ... 22222 → 9418 [ACK] Seq=11101 Ack=1 Win=8192 Len=1452 [TCP segment of a reassembled PDU]
   12   ... [TCP Retransmission] 22222 → 9418 [ACK] Seq=11101 Ack=1 Win=8192 Len=1452

With the patch, the following:

   11   ... 22222 → 9418 [ACK] Seq=11101 Ack=1 Win=8192 Len=1452 [TCP segment of a reassembled PDU]
   12   [TCP Retransmission] 22222 → 9418 [ACK] Seq=11101 Ack=1 Win=8192 Len=1452 [TCP segment of a reassembled PDU]

Georg Müller said:

(In reply to Peter Wu from comment #9)  
> I was able to reassemble stuff by skipping the return here, but more work is  
> necessary to avoid calling dissectors multiple times for the same segment.  
>  

If a dissector consumes the bytes, wouldn't the segment be finished?

What I observed with the patch, the only difference is that the retransmission is flagged as TCP segment of a reassembled PDU.

Georg Müller said:

My idea ist to replace your hack with a new function:

gboolean fragment_has_new_data(table, ...)

(name is subject to change ;) )

This function will check if the fragment contains any data not already available in the buffer by looping through the list returned by lookup_fd_head().

If fragment_has_new_data() returns false, then return and stop processing.

Peter Wu said:

(In reply to Georg Müller from comment #11)  

> With the patch, the following:  
>  
>    11   ... 22222 → 9418 [ACK] Seq=11101 Ack=1 Win=8192 Len=1452 [TCP  
> segment of a reassembled PDU]  
>    12   [TCP Retransmission] 22222 → 9418 [ACK] Seq=11101 Ack=1 Win=8192  
> Len=1452 [TCP segment of a reassembled PDU]  

This second "[TCP segment of a reassembled PDU]" does not belong there. Like the comment says, it is a hack, not a solution ;)

A complication is that retransmitted TCP segments could be larger than previously  transmitted segments, at least in theory. For example, in theory one could have transmissions of two segments:

 Seq=123 Len=456
 Seq=123 Len=890

The second could be marked as retransmission, but there is new data that should not be skipped. This is a theoretical case allowed by the spec, but not necessary present in implementations, so I might (or might not) ignore that problem at this point.

To conserve memory, not all fragments are stored in the reassembly table, so just checking that is not sufficient. I think the machinery for fixing out-of-order segments can be reused for this. This feature was previously added in v2.9.0rc0-1097-gca42331437, as well as an XXX comment pointing out this bug :P

Is there any patch available for this problem? I think I'm seeing the same issue, and it is causing some problems trying to dissect HTTP traffic if there is a packet loss.

I attached a network capture that shows two HTTP requests on the same connection. I used iptables to simulate a packet loss, dropping the first packet of the first response. There is a duplicate ACK and the packet is resent.

As you can see, curl is able to read both responses successfully, but Wireshark loses context and is not able to dissect the responses.

network.pcap curl.log

assigned to @Lekensteyn

I ran into this again with Wireshark 3.4.3. In my case the retransmission fully covers the previous segment:

No. Time        seq     nxtseq  ACK Info
13  0.355975    2522    2522    440 443 → 53488 [ACK] Seq=2522 Ack=440 Win=67584 Len=0
14  0.355979    2593    2593    587 [TCP Previous segment not captured] 443 → 53488 [ACK] Seq=2593 Ack=587 Win=68608 Len=0
15  0.665456    2522    2593    587 [TCP Retransmission] 443 → 53488 [PSH, ACK] Seq=2522 Ack=587 Win=68608 Len=71

For @johan.blumenberg's network.pcap, the retransmitted segment is larger and overlaps the previous segment too:

No. Time        seq     nxtseq  ACK Info
5   0.000005    1       1       85  80 → 51436 [ACK] Seq=1 Ack=85 Win=65535 Len=0
6   0.000005    1449    2254    85  [TCP Previous segment not captured] 80 → 51436 [PSH, ACK] Seq=1449 Ack=85 Win=65535 Len=805 [TCP segment of a reassembled PDU]
8   0.800427    1       1461    85  [TCP Retransmission] 80 → 51436 [ACK] Seq=1 Ack=85 Win=65535 Len=1460
11  0.800430    2254    2254    169 80 → 51436 [ACK] Seq=2254 Ack=169 Win=65535 Len=0

mentioned in issue #17119 (closed)

An out-of-order packet incorrectly detected as retransmission breaks desegmentation of TCP stream

Child items 0

Activity