Raw binary protocol analysis with Wireshark

I’m currently trying to analyze a binary protocol between 2 devices, but their communication does not occur over the network, neither can it be sniffed easily. Since this involves communication between 2 parties, I think the most apt software for analyzing such “conversations” would be Wireshark.

Wireshark allows for custom protocol dissectors. Writing such a dissector is usually done in C for speed, but I didn’t really want to setup the whole compilation environment to compile Wireshark. Fortunately, the Wireshark (Windows) binaries are compiled with Lua scripting support, which can also be used to write dissectors (although they run slower than C implementations).

I assume you already know how to write a simple “hello world” type dissector and you have gotten it to work in Wireshark. If you have not already done so, check out this set of presentation slides first: http://www.cacetech.com/sharkfest.09/DT06_Bjorlykke_Lua%20Scripting%20in%20Wireshark.pdf

Dump file format

In order for Wireshark to open a file that contains the conversation, the file format needs to be supported (or again you can roll your own). I chose to write the binary packets using the libpcap capture file format simply because the file format is very very simple – there’s a single header at the start of the file, and a header at the start of every packet. As a plus, endianness is automatically handled during decoding (using the header magic field), so no endian conversion takes place during recording of packets.

Linking the dissector to the data link type (DLT)

The tricky part was getting Wireshark to use the dissector that I wrote, especially since the packets are not encapsulated in a frame format like Ethernet.

The HowToDissectAnything wiki entry contains the key to linking the frame format and the dissector. It describes the use of text2pcap and the frame format USER0 – USER15. Thus the capture program needs to set the data link type of the pcap file to USER0 (147) and the dissector needs to be registered under the same link layer type. This can be done in the Lua dissector like so:

my_proto = Proto("my_proto", "My Protocol")
wtap_encap = DissectorTable.get("wtap_encap")
wtap_encap:add(wtap.USER0, my_proto)

USER0 – USER15 should be used only for temporary purposes. For long term purposes, you should obtain a DLT from the Wireshark team and perhaps submit your dissector to be included into the main tree.

Lua dissector examples

These are some nice examples to get you started on your own dissector. They range from the simplest to the more advanced.

The end result is a dissector that decodes the raw binary packets that are not encapsulated in an Ethernet frame, and easily ported from Windows to Linux to Mac without re-compilation of Wireshark.

dissector for AIM/ICQ OFT protocol
This entry was posted in code.

6 comments on “Raw binary protocol analysis with Wireshark

  1. bah says:

    Stupid. Why can’t libpcap just dump out DLT_USER* files using the default APIs. Why do they force people to re-implement pcap file formats?

  2. Bennie says:

    This is a topic that is near to my heart… Take care! Where are your contact
    details though?

  3. My brother suggested I might like this blog. He used to be entirely right.
    This post actually made my day. You cann’t believe simply
    how much time I had spent for this information! Thanks!

  4. Hermelinda says:

    Howdy! I’m at work surfing around your blog from my new apple iphone!

    Just wanted tto say I love reading your blog and
    look forward to alll your posts! Keep up the fantastic work!

  5. Roman Romanovskiy says:

    may be
    wtap_encap_table = DissectorTable.get(“wtap_encap”)
    wtap_encap_table:add(wtap.USER0, p_foo)

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

This site uses Akismet to reduce spam. Learn how your comment data is processed.