Capture filters not work when capturing from named pipes or stdin

Mahdi M. Hossaini said:

Build Information: Version 0.99.6a (SVN Rev 22276)

Copyright 1998-2007 Gerald Combs <gerald@‍wireshark.org> and contributors. This is free software; see the source for copying conditions. There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.

Compiled with GTK+ 2.10.12, with GLib 2.12.12, with WinPcap (version unknown), with libz 1.2.3, with libpcre 6.4, with Net-SNMP 5.4, with ADNS, with Lua 5.1, with GnuTLS 1.6.1, with Gcrypt 1.2.3, with MIT Kerberos, with PortAudio PortAudio V19-devel, with AirPcap.

Running on Windows XP Service Pack 2, build 2600, with WinPcap version 4.0.1 (packet.dll version 4.0.0.901), based on libpcap version 0.9.5, without AirPcap.

Built using Microsoft Visual C++ 6.0 build 8804

When we define some capture filters, and want to capture from named pipes (on Windows) the capture filter doesn't work and all received packets are captured.

added clidumpcap version2.0 labels

Guy Harris said:

This isn't unique to Windows.

I assume the theory was that the program writing to the pipe should be doing the filtering; that means you don't have to push uninteresting packets through the pipe (pushing them through the pipe is a waste of CPU and memory bandwidth, at minimum, and if the program writing to the pipe is receiving the packets from over a network - e.g., a tcpdump/WinDump run on a remote machine via ssh - it also wastes network bandwidth; in addition, it increases the risk of packet loss).

If it's impossible to do the filtering in the program writing to the pipe, we could have the capture loop run bpf_filter() on the incoming packets when reading from a pipe.

Guy Harris said:

...and if we're not going to use the filter, we shouldn't let the user specify it, as having a capture filter box fools the user into thinking it makes a difference.

As such, doing bpf_filter() on the packets when reading from a pipe is probably the right idea, if doable. HOWEVER, doing the filtering in the program writing to the pipe (or, if that program is getting its input from another program, e.g. a tcpdump/WinDump run over ssh, doing it in that program - i.e., doing itas early as possible in the pipeline) is STILL a good idea, as per my previous comment.

tenebual said:

the same problem happened on version 1.8.2

Jeff Morriss said:

*** Bug #8263 (closed) has been marked as a duplicate of this bug. ***

Tore Anderson said:

(In reply to comment #2)

...and if we're not going to use the filter, we shouldn't let the user specify it, as having a capture filter box fools the user into thinking it makes a difference.

As such, doing bpf_filter() on the packets when reading from a pipe is probably the right idea, if doable. HOWEVER, doing the filtering in the program writing to the pipe (or, if that program is getting its input from another program, e.g. a tcpdump/WinDump run over ssh, doing it in that program - i.e., doing itas early as possible in the pipeline) is STILL a good idea, as per my previous comment.

The reason I came across this issue was that I needed to dump packets on my home gateway, an embedded device with a stripped down Linux operating system on it. It has tcpdump installed, but it is compiled without IPv6 decoding support (it can only match the IPv6 ethertype, nothing else). I need to save only DHCPv6 IPv6 traffic, dumping all of it would fill my disks quite rapidly.

So I wanted to do was on the home gateway something like:

$ tcpdump -i wan0 -U -w - ip6 | nc my-workstation 12345

And on the workstation do something like:

$ nc -l 12345 | tshark -i - -f "udp and (port dhcpv6-client or port dhcpv6-server)" -w dhcpv6.pcap

In other words, thark is the earliest place in the pipeline I could apply the capture filter. But, because it ignores the capture filter, the dhcpv6.pcap quickly became much too large, since it contained all IPv6 traffic, not only DHCPv6.

I managed to find a workaround, though, by giving the filter "ip6 and ip6[6] == 17 and (ip6[40:2] == 546 or ip6[42:2] == 546 or ip6[40:2] == 547 or ip6[42:2] == 547)" to the tcpdump on the home gateway. But it would have been much easier if tshark could do it instead.

Chad M said:

Still an issue in 2.0.2, at least on OS X.

As such, doing bpf_filter() on the packets when reading from a pipe is probably the right idea, if doable. HOWEVER, doing the filtering in the program writing to the pipe (or, if that program is getting its input from another program, e.g. a tcpdump/WinDump run over ssh, doing it in that program - i.e., doing itas early as possible in the pipeline) is STILL a good idea, as per my previous comment.

I agree, doing it as early as feasibly possible in the pipeline is a good idea. There are cases though where filtering at capture time is not an option. For example, a forensic system that is capturing all network data. Or an unfiltered capture file that was taken by someone else. I deal with both, but the main pain point is trying to do analysis on forensic data. In that case the work pipeline: forensic capture -> forensic export -> local analysis.

Forensic captures, by design, are unfiltered, so filtering in the capture isn't an option. Similarly, when working with an existing capture file it isn't possible to change the capture options that were used at the time.

Next stage would be the forensic export. This is the prime spot to filter -if- you know what can safely be excluded or included. Often that is the case, but occasionally need to see multi-system behavior. That means taking a look at the unfiltered data to get an idea of what can be excluded. Exclude it, reanalyze, and repeat the process to try to whittle down the data set size.

That would be great, except analysis on the forensic system itself isn't exactly fast and what information the analysis does provide is inferior to the information available in local tools. That means grabbing the entire data set, splitting apart into manageable chunks with editcap, then start taking a look at different chunks in parallel to see what can be safely excluded.

At that point I already have a complete copy of the data so the fastest, and most efficient, thing to do would be to use the local data file as a source and filter it. I tried that repeatedly with dumpcap and tshark. Each finished processing the local source file in about 8 minutes, far less time than it would take to try to filter and export from the forensic system again. The only problem is that, due to this bug, dumpcap and tshark simply gave a copy of the original source file instead of a filtered, reduced file.

A posible alternative, at least for small data sets / source files, would be to use a display filter in Wireshark and then save only the displayed packets to a new file. When the source file is more than a few hundred MB though the time required to both open and filter the file is very significant. When working with multi-GB source files (12.5 GB / 25 million packets most recently) that method simply isn't a viable alternative. Instead, that is where dumpcap / tshark come into play. They should be able to take an input file (stdin / pipe / socket), apply a filter, and write the filtered data to an output file, which is exactly what the man pages imply should be happening already.

Christopher Maynard said:

This bug seems more relevant to the capture process, not the UI, so changing the component to Dumpcap instead of GTK+ UI.

Guy Harris said:

Note also that the filter cannot be tested in the input box, as you don't know what link-layer header type(s) will be coming down the pipe.

If we support piping arbitrary pcapng files to dumpcap etc., it's more complicated, as there may be more than one link-layer type.

(Then again, I'm going to have to solve that problem when I add full pcapng support to libpcap; I may be able to use a similar scheme here. It may involve parsing a filter expression into a parse tree or some link-layer-header-independent form, and then generating code from that. We might then treat filter expressions that test something not available with some particular link-layer header type as warnings, rather than errors, and just have the filter reject all packets, e.g. if you're reading a file with Ethernet packets and 802.11 packets, an expression that checks for a WLAN type other than data will provoke a warning as soon as we determine that there may be Ethernet packets in the capture.)

In addition, even for pcap files, the link-layer header type can't be determined until we read the header from the pipe. (That might be solvable with the same mechanism.)

abujed said:

Please boot me to where this needs to go if I've posted it in the wrong place. I looked but was unable to find my issue anywhere else, so posting it here:

Capture filters on the tshark command line do not appear to be working in Kali-rolling linux on ARM (Raspberry Pi 3 B+) and Panda Wireless PAU05.

I have not yet tested for this behavior on other platforms or with a different wireless adapter.

See traces below:

Now it works...

root@kali:~# iwconfig wlan1
wlan1     IEEE 802.11  Mode:Monitor  Frequency:2.412 GHz  Tx-Power=20 dBm
          Retry short  long limit:2   RTS thr:off   Fragment thr:off
          Power Management:off

root@kali:~# tshark -i wlan1 -f "wlan type mgt && wlan subtype probe-req or wlan subtype beacon"
Running as user "root" and group "root". This could be dangerous.
Capturing on 'wlan1'
    1 0.000000000 <mac obfuscated> → Broadcast    802.11 286 Beacon frame, SN=3875, FN=0, Flags=........, BI=100, SSID=<obfuscated>
    2 0.007591368 <mac obfuscated> → Broadcast    802.11 292 Beacon frame, SN=2581, FN=0, Flags=........, BI=100, SSID=<obfuscated>
    3 0.102339803 <mac obfuscated> → Broadcast    802.11 286 Beacon frame, SN=3876, FN=0, Flags=........, BI=100, SSID=<obfuscated>


root@kali:~# tshark -v
Running as user "root" and group "root". This could be dangerous.
TShark (Wireshark) 2.6.8 (Git v2.6.8 packaged as 2.6.8-1)

Copyright 1998-2019 Gerald Combs <gerald@wireshark.org> and contributors.
License GPLv2+: GNU GPL version 2 or later <http://www.gnu.org/licenses/old-licenses/gpl-2.0.html>
This is free software; see the source for copying conditions. There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.

Compiled (64-bit) with libpcap, with POSIX capabilities (Linux), with libnl 3,
with GLib 2.58.3, with zlib 1.2.11, with SMI 0.4.8, with c-ares 1.14.0, with Lua
5.2.4, with GnuTLS 3.6.7, with Gcrypt 1.8.4, with MIT Kerberos, with MaxMind DB
resolver, with nghttp2 1.37.0, with LZ4, with Snappy, with libxml2 2.9.4.

Running on Linux 4.19.29-Re4son-v8+, with 917 MB of physical memory, with locale
en_US.UTF-8, with libpcap version 1.8.1, with GnuTLS 3.6.6, with Gcrypt 1.8.4,
with zlib 1.2.11, binary plugins supported (13 loaded).

Built using gcc 8.3.0.

root@kali:~# uname -a
Linux kali 4.19.29-Re4son-v8+ #6 SMP PREEMPT Wed Mar 27 00:15:50 UTC 2019 aarch64 GNU/Linux

root@kali:~# cat /etc/*release
DISTRIB_ID=Kali
DISTRIB_RELEASE=kali-rolling
DISTRIB_CODENAME=kali-rolling
DISTRIB_DESCRIPTION="Kali GNU/Linux Rolling"
PRETTY_NAME="Kali GNU/Linux Rolling"
NAME="Kali GNU/Linux"
ID=kali
VERSION="2019.2"
VERSION_ID="2019.2"
ID_LIKE=debian
ANSI_COLOR="1;31"
HOME_URL="https://www.kali.org/"
SUPPORT_URL="https://forums.kali.org/"
BUG_REPORT_URL="https://bugs.kali.org/"

Now it doesn't work... :-(

root@kali:~# iwconfig wlan1
wlan1     IEEE 802.11  Mode:Monitor  Frequency:5.17 GHz  Tx-Power=31 dBm
          Retry short limit:7   RTS thr:off   Fragment thr:off
          Power Management:on


********undesirable behavior starts here*********
root@kali:~# tshark -i wlan1 -f "wlan type mgt && wlan subtype probe-req or wlan subtype beacon"
Running as user "root" and group "root". This could be dangerous.
Capturing on 'wlan1'
tshark: Invalid capture filter "wlan type mgt && wlan subtype probe-req or wlan subtype beacon" for interface 'wlan1'.

That string isn't a valid capture filter (802.11 link-layer types supported only on 802.11).
See the User's Guide for a description of the capture filter syntax.
0 packets captured
********undesirable behavior ends here*********



root@kali:~# tshark -v
Running as user "root" and group "root". This could be dangerous.
TShark (Wireshark) 2.6.9 (Git v2.6.9 packaged as 2.6.9-1)

Copyright 1998-2019 Gerald Combs <gerald@wireshark.org> and contributors.
License GPLv2+: GNU GPL version 2 or later <http://www.gnu.org/licenses/old-licenses/gpl-2.0.html>
This is free software; see the source for copying conditions. There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.

Compiled (64-bit) with libpcap, with POSIX capabilities (Linux), with libnl 3,
with GLib 2.58.3, with zlib 1.2.11, with SMI 0.4.8, with c-ares 1.14.0, with Lua
5.2.4, with GnuTLS 3.6.7, with Gcrypt 1.8.4, with MIT Kerberos, with MaxMind DB
resolver, with nghttp2 1.37.0, with LZ4, with Snappy, with libxml2 2.9.4.

Running on Linux 4.19.55-Re4son-v8+, with 917 MB of physical memory, with locale
en_US.UTF-8, with libpcap version 1.8.1, with GnuTLS 3.6.7, with Gcrypt 1.8.4,
with zlib 1.2.11, binary plugins supported (13 loaded).

Built using gcc 8.3.0.
root@kali:~# uname -a
Linux kali 4.19.55-Re4son-v8+ #1 SMP PREEMPT Sun Jun 30 10:40:44 UTC 2019 aarch64 GNU/Linux

root@kali:~# cat /etc/*release
DISTRIB_ID=Kali
DISTRIB_RELEASE=kali-rolling
DISTRIB_CODENAME=kali-rolling
DISTRIB_DESCRIPTION="Kali GNU/Linux Rolling"
PRETTY_NAME="Kali GNU/Linux Rolling"
NAME="Kali GNU/Linux"
ID=kali
VERSION="2019.2"
VERSION_ID="2019.2"
ID_LIKE=debian
ANSI_COLOR="1;31"
HOME_URL="https://www.kali.org/"
SUPPORT_URL="https://forums.kali.org/"
BUG_REPORT_URL="https://bugs.kali.org/"

Guy Harris said:

(In reply to abujed from comment #9)

Please boot me to where this needs to go if I've posted it in the wrong place.

Try this with tcpdump - you are 99.99999999999999999999999999% sure to get similar results - and then go to https://github.com/the-tcpdump-group/libpcap/issues and open a new issue.

This bug is for a completely separate issue, involving capturing from pipes, not from capture devices; your issue is with capturing from capture devices, and the issue is 99.99999999999999999999999999% to either be a libpcap issue, a Linux kernel issue, or a "the way wlan1 is set up" issue.

mentioned in issue #8263 (closed)

tcpdump can apply a capture filter when reading a file. Would be nice if there was a Wireshark binary that supported this for testing capture filters.

(vlan.cap is available from the Wiki Sample Captures)

$ tcpdump -r ./vlan.cap -q | wc
reading from file ./vlan.cap, link-type EN10MB (Ethernet)
    395    3835   46462

$ tcpdump -r ./vlan.cap -q port 1162 | wc
reading from file ./vlan.cap, link-type EN10MB (Ethernet)
      0       0       0

$ tcpdump -r ./vlan.cap -q vlan | wc
reading from file ./vlan.cap, link-type EN10MB (Ethernet)
    389    3761   45776

$ tcpdump -r ./vlan.cap -q vlan and port 1162 | wc
reading from file ./vlan.cap, link-type EN10MB (Ethernet)
    139     973    9403

ICMPv6 capture filter fails for MLD messages
Ugh - bump back up in the issues stack

Capture filters not work when capturing from named pipes or stdin

Child items ...

Activity