I’m currently trying to analyze a binary protocol between 2 devices, but their communication does not occur over the network, neither can it be sniffed easily. Since this involves communication between 2 parties, I think the most apt software for analyzing such “conversations” would be Wireshark.

Wireshark allows for custom protocol dissectors. Writing such a dissector is usually done in C for speed, but I didn’t really want to setup the whole compilation environment to compile Wireshark. Fortunately, the Wireshark (Windows) binaries are compiled with Lua scripting support, which can also be used to write dissectors (although they run slower than C implementations).

I assume you already know how to write a simple “hello world” type dissector and you have gotten it to work in Wireshark. If you have not already done so, check out this set of presentation slides first: http://www.cacetech.com/sharkfest.09/DT06_Bjorlykke_Lua%20Scripting%20in%20Wireshark.pdf

Dump file format

In order for Wireshark to open a file that contains the conversation, the file format needs to be supported (or again you can roll your own). I chose to write the binary packets using the libpcap capture file format simply because the file format is very very simple – there’s a single header at the start of the file, and a header at the start of every packet. As a plus, endianness is automatically handled during decoding (using the header magic field), so no endian conversion takes place during recording of packets.

Linking the dissector to the data link type (DLT)

The tricky part was getting Wireshark to use the dissector that I wrote, especially since the packets are not encapsulated in a frame format like Ethernet.

The HowToDissectAnything wiki entry contains the key to linking the frame format and the dissector. It describes the use of text2pcap and the frame format USER0 – USER15. Thus the capture program needs to set the data link type of the pcap file to USER0 (147) and the dissector needs to be registered under the same link layer type. This can be done in the Lua dissector like so:

my_proto = Proto("my_proto", "My Protocol")
...
wtap_encap = DissectorTable.get("wtap_encap")
wtap_encap:add(wtap.USER0, my_proto)
...

USER0 – USER15 should be used only for temporary purposes. For long term purposes, you should obtain a DLT from the Wireshark team and perhaps submit your dissector to be included into the main tree.

Lua dissector examples

These are some nice examples to get you started on your own dissector. They range from the simplest to the more advanced.

• dissector for AIM/ICQ OFT protocol
  very clean and simple code – use this to get started
• LLUDP dissector
• Battle.net protocol dissector

The end result is a dissector that decodes the raw binary packets that are not encapsulated in an Ethernet frame, and easily ported from Windows to Linux to Mac without re-compilation of Wireshark.

dissector for AIM/ICQ OFT protocol