This is several questions in one.
The best way to feed Wireshark with data using a pipe is to use the extcap API, which allows Wireshark to learn the properties of the capturing interface, packet format etc. and indicate to the capturing process which pipe to use. You can do that manually as well but it is not convenient for repeated use.
Regardless how you start feeding Wireshark through a pipe, the data fed to the pipe have to follow the pcap or, which is a recent improvement, pcapng format.
And regardless whether you encapsulate your application data into a pcap(ng) record directly or whether they are encapsulated in e.g. USB URBs, you need a piece of software called dissector to convert the binary data of a given protocol into a human-readable form. The dissector needs to be pointed to the portion of data it is responsible for, which is usually accomplished by means of so dissector tables.
The instruction how to write a dissector and hook it to the right place of the dissector tree can be found here. You can also use the Lua API for Wireshark for rapid prototyping of dissectors - look here.
Specially for generic analysis of serial connections, it makes sense to make the transport layer PDU (Protocol Data Unit) a single byte, because it is the only way how to provide each byte with its own timestamp. If you do that, you'll likely have to use reassembly to complete the application layer PDUs (like e.g. modem AT commands).
Also, if serial connections are what you are really interested in, you may use virtual serial ports linked to Serial over LAN adaptors and capture the IP communication between your machine and the adaptor.
Hey! I am trying something similar.. Could you please tell how you read the COM port data to the named pipe? Also, did you use pcap header format?
Thanks.