Filtering tshark captures with read filters (-R) no longer works
This issue was migrated from bug 2234 in our old bug tracker.
Original bug information:
Reporter: Pete Christensen
Status: CONFIRMED
Product: Wireshark
Component: TShark
OS: Windows XP
Platform: x86
Version: 0.99.7
Attachments:
trace-raw.pcap: example trace file
Activity
Build Information: Could not open file: 'Ericsson.xml', error: No such file or directory TShark 0.99.7 (SVN Rev 23910)
Copyright 1998-2007 Gerald Combs <gerald@wireshark.org> and contributors. This is free software; see the source for copying conditions. There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.
Compiled with GLib 2.14.3, with WinPcap (version unknown), with libz 1.2.3, with
libpcre 6.4, with SMI 0.4.5, with ADNS, with Lua 5.1, with GnuTLS 1.6.1, with Gcrypt 1.2.3, with MIT Kerberos.
Running on Windows XP Service Pack 2, build 2600, with WinPcap version 4.0.2 (packet.dll version 4.0.0.1040), based on libpcap version 0.9.5.
Built using Microsoft Visual C++ 6.0 build 8804
Use a -R display filter with tshark while capturing to a file. For example: tshark -i 2 -w snork.cap -R "ip.addr == 192.168.1.132" 99.6a: snork.cap contains ONLY those packets filtered in by the display filter. 99.7: The packet count increments based on filtered-in packets, but snork.cap actually contains ALL packets seen on the interface, not just packets filtered in by the display filter. Note that I've priority to high since this basically breaks the previous functionality which is very useful. Order of the -R argument does not seem to make a difference. I didn't test this on other released packages for other systems, only Windows.added clitshark oswindows version0.x labels
-
Yes, this is actually the expected behavior now. As a result of the work done for privilege separation the (small, reasonable to run setuid-root) tool doing the capturing (dumpcap) is also writing out the files. tshark then reads that file in so it can display the output (when not using "-w" to write the file). When using "-w" to write the file, well, tshark actually doesn't do anything but print statistics(?). There was a lot of discussion about this at the time but, IIRC, the consensus(?) basically was: - we absolutely needed priv sep (Wireshark has way too much code to all be running as root, especially since it's looking at potentially malicious network traffic) - it's more important that dumpcap be fast so that it doesn't drop packets than to keep read filters while capturing to a file (that's why it writes the files directly rather than trying to send the packets to Wireshark/tshark through a pipe) Probably due to the controversy of some of that we never got around to preventing "-R" from working with "-w". I suppose this bug should at least do that. -
*** Bug #2776 (closed) has been marked as a duplicate of this bug. ***
-
-w -R works fine if you are reading from a file (-r). The problem here is that if you're capturing megabytes of data and try to filter this on the fly to make a smaller resulting file then this is not possible anymore, and this is problematic for me too.
I understand why -w is ignored, but this results in basically ignoring -R in tshark when not reading from a file and forces people to use more disk space and make the analysis a two step (two files) process.
-
We could, I guess, have tshark, when run with "-w" and "-R", have dumpcap pipe the capture to it rather than writing the capture to a file, and have tshark read from the pipe, dissect the packets, apply the read filter, and write the packets that match to the file. (Yes, this might slow capture down a bit, but we will not do dissection in any process that needs to run with privileges - that is non-negotiable - which means that either 1) there need to be two programs/processes involved or 2) this feature would only work on platforms that don't require dangerously elevated privileges to capture - such platforms exist, but they need to be configured to allow that or 3) we'd have to pull the capture code back into tshark and have it start the capture first and then give up privileges, which is a bit complicated and ugly.) See also bug 2743. -
The wireshark man page reads as follows: -R <read (display) filter> When reading a capture file specified with the -r flag, causes the specified filter (which uses the syntax of display filters, rather than that of capture filters) to be applied to all packets read from the capture file; packets not matching the filter are dis‐ carded. The tshark man page reads as follows: -R <read (display) filter> Cause the specified filter (which uses the syntax of read/display filters, rather than that of capture filters) to be applied before printing a decoded form of packets or writing packets to a file; packets not matching the filter are discarded rather than being printed or written. I think this is a tshark documentation bug. I suggest syncing the tshark and wireshark man pages for the -R flag documentation. Probably the original behaviour of read filters applying to captures was wrong to begin with. -
*** Bug #2872 (closed) has been marked as a duplicate of this bug. ***
-
Hi again, this problem is driving me crazy, my capture files are gigantic. I can't use capture filters (see bug 2804) to reduce the size. So what about getting the behavior back to what it was when tshark reads from a pipe? Something like: tshark -i wlan1 .... -w- | tshark -i- -R (filter) -w capture The second tshark would see it's reading from a pipe and would then apply the read filters ? Or maybe a new option to get the behavior back to what it was? Thanks! matt -
*** Bug #4038 (closed) has been marked as a duplicate of this bug. ***
-
(In reply to comment #11) > All, > > So what is the resolution? How do I get the same functionality as the previous > tshark version? > > Is this working in the newer release? The bug is still open, so there is no resolution yet. One workaround is, as mentioned, to do the display filtering in a post-capture run, though obviously that doesn't help you avoid writing all those unwanted packets to the disk in the first place. -
(In reply to comment #12) > (In reply to comment #11) > > > > So what is the resolution? How do I get the same functionality as the previous > > tshark version? > > > > Is this working in the newer release? > > The bug is still open, so there is no resolution yet. > > One workaround is, as mentioned, to do the display filtering in a post-capture > run, though obviously that doesn't help you avoid writing all those unwanted > packets to the disk in the first place. Another workaround is to transform the display filter to a capture filter. The command from your initial big-report could be transformed into: tshark -f "(<have a long string of IP's>) and (udp port 5060 or udp port 53 or tcp port 3868)" -F libpcap -w Trace -a duration:30 -
Hi,
I guess there will never be a fix for this issue.
The display filter works fine when reading from a capture file and output to the screen but not when writing (-w) to another file. The data in the new file is incorrect yet the IP and TCP (sequence and acknowledge numbers) information seem the same. Is this the same bug?
The commend in use is tshark -r <file1> -w <file2> -R http.response.code==400
Thx Mark
-
(In reply to comment #14) > The display filter works fine when reading from a capture file and output to > the screen but not when writing (-w) to another file. The data in the new file > is incorrect yet the IP and TCP (sequence and acknowledge numbers) information > seem the same. Is this the same bug? > > The commend in use is tshark -r <file1> -w <file2> -R http.response.code==400 I use "tshark -r <infile> -w <outfile> -R <filter>" a lot and it should work properly. What do you mean by "The data in the new file is incorrect"? Could it be that the http response was actually reassembled (ie the response was split over more than 1 packet on the network)? If so, you are hitting the enhancement request filed in bug 3315. -
(In reply to comment #16)
> Created an attachment (id=3848) [details]
> example trace fileSorry I lost the reply in my attachment. The file attached indicates the issue, the response is reassambled yet the line text with a 400 message is dropped. Guess there is no way round this
-
(In reply to comment #17) > > Sorry I lost the reply in my attachment. The file attached indicates the issue, > the response is reassambled yet the line text with a 400 message is dropped. > Guess there is no way round this Well, when reassembly is enabled, only the packet with the last segment will have upper-layer dissection and only this packet will be displayed and/or saved. When the first segment contains enough information to make the display filter work, then you could disable reassembly. Of course you will get only the first TCP segment of the HTTP responses, but it looks like this is the info you seek. Here is how you can do it in tshark: $ tshark r trace-raw.pcap -R http.response.code==400 14 0.161606 10.66.27.175 -> 10.76.16.1 HTTP HTTP/1.1 400 Bad request (text/html) $ tshark -o tcp.desegment_tcp_streams:FALSE -r trace-raw.pcap -R http.response.code==400 7 0.062142 10.66.27.175 -> 10.76.16.1 HTTP HTTP/1.1 400 Bad request As you can see, now frame 7 (in which the first part of the response is present) is displayed instead of frame 14 (which contains the last part of the response). -
*** Bug #3518 (closed) has been marked as a duplicate of this bug. ***
-
*** Bug #6681 (closed) has been marked as a duplicate of this bug. ***
-
From r43102 it's possible to use pipe as workaround.
Instead of: $ tshark -i XXX -R 'yourfilter' -w file.pcap do: $ dumpcap -i XXX -w - | tshark -r - -R 'yourfilter' -w file.pcap
It's worthy to mention that wiretap don't have any special handling of pipes, and it'll read data in whole I/O block size for pipe (typically 4096B). Captured data will be saved with delay, and you might also lose last 4K of data.
-
(In reply to comment #22) > From r43102 it's possible to use pipe as workaround. > > Instead of: > $ tshark -i XXX -R 'yourfilter' -w file.pcap > do: > $ dumpcap -i XXX -w - | tshark -r - -R 'yourfilter' -w file.pcap > > It's worthy to mention that wiretap don't have any special handling of pipes, > and it'll read data in whole I/O block size for pipe (typically 4096B). > Captured data will be saved with delay, and you might also lose last 4K of > data. undefine@uml:~$ sudo dumpcap -i wlan0 -w - | tshark -r - -R 'sip' -w file.pcap File: - Packets: 59 tshark: The file "-" could not be opened: Illegal seek. Packets: 62 Packets dropped: 0 doesn't work on all tshark versions. here i have 1.4. I just backported tshark from 0.99 to lenny and squeeze and it works fine :) -
(In reply to comment #23) > undefine@uml:~$ sudo dumpcap -i wlan0 -w - | tshark -r - -R 'sip' -w file.pcap > File: - > Packets: 59 tshark: The file "-" could not be opened: Illegal seek. > Packets: 62 Packets dropped: 0 > > doesn't work on all tshark versions. here i have 1.4. For information r43102 is part of the 1.8.X branch so you need a new Wireshark version. -
OK, the situation is less than stellar, maybe we need to get back to allowing -R the way it was: tshark -i utun2 -R "ip.addr==10.122.4.12" tshark: -R without -2 is deprecated. For single-pass filtering use -Y. tshark -i utun2 -Y "ip.addr==10.122.4.12" Capturing on 'utun2' ... ^C4 packets captured tshark -w test.pcapng -i utun2 -Y "ip.addr==10.122.4.12" tshark: Display filters aren't supported when capturing and saving the captured packets. tshark -w test.pcapng -i utun2 -R "ip.addr==10.122.4.12" tshark: -R without -2 is deprecated. For single-pass filtering use -Y. tshark -w test.pcapng -i utun2 -R "ip.addr==10.122.4.12" -2 tshark: Live captures do not support two-pass analysis. -
(In reply to Jörg Mayer from comment #25)
> OK, the situation is less than stellar, maybe we need to get back to
> allowing -R the way it was:Jakub's suggestion from comment #22 works, but isn't ideal.
Breaking the privilege separation model is untenable.
Ideally perhaps dumpcap (or extcap tools) should be allowed to pipe data to *shark instead of writing a temp file. The *shark can run a display filter without privileges, and write the filtered data to a file?
-
(In reply to Stephen Donnelly from comment #26)
> Ideally perhaps dumpcap (or extcap tools) should be allowed to pipe data to
> *shark instead of writing a temp file. The *shark can run a display filter
> without privileges, and write the filtered data to a file?Yes. That's basically how *shark worked before privsep.
That is, back then *shark read the data, ran the read filter, and wrote the results to a file. Now dumpcap reads the data, writes the file, and *shark reads the file (and applies display filters).
-
(In reply to Jeff Morriss from comment #27) > Yes. That's basically how *shark worked before privsep. > > That is, back then *shark read the data, ran the read filter, and wrote the > results to a file. Now dumpcap reads the data, writes the file, and *shark > reads the file (and applies display filters). The difference proposed is that dumpcap/extcap (privileged) captures the data, but *shark (unprivileged) does the filtering and disk writing. That should maintain privsep? -
(In reply to Stephen Donnelly from comment #28) > (In reply to Jeff Morriss from comment #27) > > > Yes. That's basically how *shark worked before privsep. > > > > That is, back then *shark read the data, ran the read filter, and wrote the > > results to a file. Now dumpcap reads the data, writes the file, and *shark > > reads the file (and applies display filters). > > The difference proposed is that dumpcap/extcap (privileged) captures the > data, but *shark (unprivileged) does the filtering and disk writing. That > should maintain privsep? Yes: the vast majority of the full-stack code would still be running as a non-privileged user (in fact it moves the act of writing the file from a privileged user to a non-privileged one) so it maintains privsep. mentioned in issue #2776 (closed)
mentioned in issue #2804 (closed)
mentioned in issue #2872 (closed)
mentioned in issue #3518 (closed)
mentioned in issue #4038 (closed)
mentioned in issue #6681 (closed)
removed version0.x label
One of the minor annoyances with this change is if you run a lot of
-zstatistics, and you want a single filter that applies to all of them. Read filters (-R) apply to statistics, but display filters (-Y) do not. You end up having to do a two pass tshark in order to have a filter that applies to every statistic, or else copy the same long filter line everywhere.The tshark man page claims:
Read filters can be specified when capturing or when reading from a capture file. Note that capture filters are much more efficient than read filters, and it may be more difficult for TShark to keep up with a busy network if a read filter is specified for a live capture, so you might be more likely to lose packets if you’re using a read filter.But in fact, as noted above, this is not the case. Neither read filters nor display filters can be specified when capturing and writing to a file, and when capturing and only displaying a file, only a display filter (not a read filter) be specified, since two-pass analysis is not allowed.
Edited by John Thackermentioned in merge request !10066
