ipsec: refactor the decryption and dissection of ESP packets

Introduction

This merge request started as a simple attempt to fix the decryption and dissection of AES-GCM encrypted ESP packages: The ipsec dissector did not strip the ICV from the decrypted payload correctly, in fact it didn't consider the ICV length at all. As a consequence, it failed to recognize the protocol type of the decrypted packet correctly (which is stored in the last byte of the decrypted payload). Most of the times it would find an invalid protocol number and simply show a raw hex dump, sometimes it would find a random protocol and dissect it as garbage.

After a while I noticed that in order to implement full blown AES-GCM decryption support (i.e., support for AEAD ciphers with verification of the ICV) it would be necessary to refactor the somewhat convoluted ipsec dissector (packet-ipsec.c) implementation following the example of the isakmp dissector (packet-isakmp.c), which was simpler and straighforward and already had a functional AES-GCM support for IKEv2. As a side effect of the refactoring, also the dissection of the existing 'classical' encryption methods with separate authentication has improved a lot.

The following screen shots demonstrate the changes by comparing the dissector old and new dissector output. For a detailed description of the changes please read the commit messages from bottom to top.

This is my first merge request for the Wireshark project, so please bear with me if I failed to respect some Coding Guidelines. Also, in some cases I am still a bit unsure how to react correctly to error conditions (throw exception, add tree item, add expert item, ...) in order to maximize the user experience. So I'm looking forward to your feedback and suggestions for improvement.

Examples

The following packet captures can be found as SampleCaptures in the Wireshark Wiki.

IPsec - Example 2: Dissection of encrypted (and UDP-encapsulated) IKEv2 and ESP messages

The screenshots below were created using the archive:

• ipsec_ikev2+esp_aes-gcm,aes-ctr,aes-cbc.tgz

A VPN client (192.168.245.131) behind a NAT device connects three times to a VPN gateway (172.16.15.92) using IKEv2, sends some pings through the VPN tunnel (192.168.225.0/24) to the gateway (192.168.225.1), which are returned successfully, and disconnects. The three connections differ by the AES operation modes (AES-GCM, AES-CTR, and AES-CBC, in that order) used for encrypting the IKE_AUTH and ESP messages:

Nr	Encryption	Authentication
1	AES-GCM with 16 octet ICV [RFC4106]	NULL
2	AES-CTR [RFC3686]	HMAC-SHA-256-128 [RFC4868]
3	AES-CBC [RFC3602]	HMAC-SHA-256-128 [RFC4868]

The entire conversation (IKE+ESP) is sent UDP-encapsulated on port 4500.

The archive contains :

• capture.pcap: the packet capture file
• esp_sa: the decryption table for the ESP SAs
• ikev2_decryption_table: the decryption table for the IKEv2 SAs

Note that to load the example with your own Wireshark copy, you either need to replace AES-GCM with 16 octet ICV [RFC4106] -> AES-GCM [RFC4106] or patch your working copy with commit d6557271.

IPsec - Example 1: ESP Payload Decryption and Authentication Checking Examples

Those historic examples from 2006 were updated by me(*) and can be used to verify that my refactoring did not break the decryption of older ciphers.

(*) Essentially, I converted the old Ethereal SA settings into Wireshark decryption tables (esp_sa), otherwise they are unchanged.

Screen Shots

AES-GCM

old

new

AES-CTR

old

new

AES-CBC

old

new