dissector or decoding for data payload

edit

There may be several ways to go about this. One route is to write a Lua script for your extensions. This uses an API exposed by Wireshark that allows additions to the dissection engine, i.e., for your protocol. Further details can be found in the Wireshark Developer's Guide.

When you're looking into coding your own dissector in C you can reference the Wireshark Developer's Guide how to setup your development environment, and find additional documentation in the /doc directory of the source tree.