Ask Your Question
0

lua example for protobuf dissector which has protobuf encoded within protobuf

asked 2024-03-09 07:50:49 +0000

Babuvel gravatar image

updated 2024-03-10 06:47:19 +0000

Hi,

Am looking for working example for below case where data is serialized as bytes to another protobuf.

In below example streamDataArgs msg includes "data" as bytes and that is serialized with encodeData.proto. This is a very simple case and I would like to know if there is simpler way to get the working lua example for this.

I found https://ask.wireshark.org/question/21... but that neither had the protofile nor the full lua script example to check.

serviceA.proto

syntax = "proto3";

package export_data;

service streamDataOut { 
    rpc streamData(stream streamDataArgs) returns(stream streamDataArgs) {}; 
}

message streamDataArgs { 
    int64 Id = 1; 
    bytes data = 2; // serialized with encodeDataInfo message from encodeData.proto 
}

encodeData.proto

syntax = "proto3";

package encoded_data;

message encodeDataInfo { 
    string name = 1; // set to macData.proto
    bytes val = 2;     // serialized with macDataInfo from macData.proto
}

macData.proto

syntax = "proto3";

package mac_data;

message macDataInfo { 
    string mac_addr = 1; // set to mac address
    uint64 time = 2;        // set to epoch time in ms
}

pcap file

https://www.dropbox.com/scl/fi/nayzxf...

edit retag flag offensive close merge delete

Comments

Can share a capture file and proto file(s)?

Chuckc gravatar imageChuckc ( 2024-03-09 13:59:57 +0000 )edit

Thanks for the response. I have added both protofiles and the pcap capture dropbox link to the question. Please let me know if more details required.

Babuvel gravatar imageBabuvel ( 2024-03-10 06:48:30 +0000 )edit

Do you have lua code for the first encoding? (serviceA.proto)

Chuckc gravatar imageChuckc ( 2024-03-12 17:51:27 +0000 )edit

no, I don't have it. Looking for help here with that.

Babuvel gravatar imageBabuvel ( 2024-03-12 18:54:57 +0000 )edit

Link back to a similar question: Protobuf dissector with nested structures

Chuckc gravatar imageChuckc ( 2024-03-13 14:34:55 +0000 )edit

3 Answers

Sort by » oldest newest most voted
0

answered 2024-03-24 20:47:35 +0000

Chuckc gravatar image

Final answer - matches protobuf described in answer 2 comments:

-- 240313_33954_nested_pb_table.lua
-- "Am looking for working example for below case where data is serialized as bytes to another protobuf."
-- https://ask.wireshark.org/question/33954/lua-example-for-protobuf-dissector-which-has-protobuf-encoded-within-protobuf/

-- Step 1 - document as you go. See header above and set_plugin_info().
local nested_pb_table_info =
{
    version = "1.0.0",
    author = "Good Coder",
    description = "Decode nested protobufs",
    repository = "Floppy in top drawer"
}

set_plugin_info(nested_pb_table_info)

do
    local protobuf_field_table = DissectorTable.get("protobuf_field") 
    local protobuf_dissector = Dissector.get("protobuf")

    -- This could probably be one dissector with more control logic
    -- It is easier (IM"not so"HO) to read split out into two dissectors
    encoded_pb_p = Proto("encoded_pb_dissector", "Encoded Protobuf")
    encoded_pb_p.dissector = function(tvb, pinfo, subtree)
        pinfo.private["pb_msg_type"] = "message,encoded_data.encodeData"
        local subsubtree = subtree:add(encoded_pb_p, tvb())
        pcall(Dissector.call, protobuf_dissector, tvb, pinfo, subsubtree)
    end

    pbf_encodeDataInfo_name_f = Field.new("pbf.encoded_data.encodeDataInfo.name")
    pbf_encodeDataInfo_val_f = Field.new("pbf.encoded_data.encodeDataInfo.val")

    netdata_pb_p = Proto("netdata_pb_dissector", "Network Data Protobuf")
    netdata_pb_p.dissector = function(tvb, pinfo, subtree)

        -- What is the message type for the nested protobufs
        local message_type = {
            ["macData.proto"] = "message,mac_data.macDataInfo",
            ["ipData.proto"] = "message,ip_data.ipDataInfo"
        }

        finfo_names = { pbf_encodeDataInfo_name_f() }
        finfo_vals = { pbf_encodeDataInfo_val_f() }

        -- the current last pbf.encoded_data.encodeDataInfo.name contains the .proto
        -- filename matching the pbf.encoded_data.encodeDataInfo.val passed in tvb
        -- use message_type table to map it message type
        if (#finfo_names > 0 and #finfo_vals >0) then
            v_string = string.format("%s", finfo_names[#finfo_names])
            pinfo.private["pb_msg_type"] = message_type[v_string]
            local subsubtree = subtree:add(netdata_pb_p, tvb())
            pcall(Dissector.call, protobuf_dissector, tvb, pinfo, subsubtree)
        end
    end

    -- What fields are the encoded protobufs in
    protobuf_field_table:add("export_data.streamDataArgs.data", encoded_pb_p) 
    protobuf_field_table:add("encoded_data.encodeDataInfo.val", netdata_pb_p) 
end

edit flag offensive delete link more
0

answered 2024-03-14 14:11:35 +0000

Chuckc gravatar image

From the Wireshark wiki - Protobuf#protocol-dependencies:

Protobuf content is normally dissected by Wireshark from some higher layer dissectors including gRPC or other UDP/TCP based dissectors

There are two TCP streams in the provided pcap file:

image description

Add a Decode As... entry for a Stream ID 1 TCP port (34727):

# "Decode As" entries file for Wireshark 4.2.3.
#
# This file is regenerated each time "Decode As" preferences
# are saved within Wireshark. Making manual changes should be safe,
# however.
decode_as_entry: tcp.port,34727,(none),HTTP2

The out of the box protobuf decode:

Frame 46: 127 bytes on wire (1016 bits), 127 bytes captured (1016 bits) on interface lo, id 0
Ethernet II, Src: 00:00:00_00:00:00 (00:00:00:00:00:00), Dst: 00:00:00_00:00:00 (00:00:00:00:00:00)
Internet Protocol Version 4, Src: 127.0.0.1, Dst: 127.0.0.1
Transmission Control Protocol, Src Port: 34727, Dst Port: 50051, Seq: 602, Ack: 231, Len: 61
HyperText Transfer Protocol 2
GRPC Message: /export_data.streamDataOut/streamData, Request
Protocol Buffers: /export_data.streamDataOut/streamData,request
    Message: export_data.streamDataArgs
        [Message Name: export_data.streamDataArgs]
        Field(1): Id = 1234 (int64)
            [Field Name: Id]
            [Field Type: int64 (3)]
            .000 1... = Field Number: 1
            .... .000 = Wire Type: varint (0)
            Value: d209
                Int64: 1234
            Id: 1234
        Field(2): data  (bytes)
            [Field Name: data]
            [Field Type: bytes (12)]
            .001 0... = Field Number: 2
            .... .010 = Wire Type: Length-delimited (2)
            Value Length: 42
            Value: 0a0d6d6163446174612e70726f746f12190a1130303a30303a30303a30303a31323a464610eb86b5af06
            data: (42 bytes)
                Data (42 bytes)
                    Data: 0a0d6d6163446174612e70726f746f12190a1130303a30303a30303a30303a31323a464610eb86b5af06
                    Text: \n\rmacData.proto\x12\x19\n\x1100:00:00:00:12:FF\x10놵�\x06
                    [Length: 42]


Wireshark wiki - Protobuf field subdissectors:

A subdissector can register itself in "protobuf_field" dissector table for parsing the value of the field. The key of record in "protobuf_field" table is the full name of the field.

    local protobuf_field_table = DissectorTable.get("protobuf_field")
    local png_dissector = Dissector.get("png")
    protobuf_field_table:add("tutorial.Person.portrait_image", png_dissector)

In their example, it's to decode a png. Our goal is to decode an embedded protobuf:

local protobuf_dissector = Dissector.get("protobuf")
...
pinfo.private["pb_msg_type"] = "message,tutorial.AddressBook"
pcall(Dissector.call, protobuf_dissector, tvb, pinfo, tree)


-- 240313_33954_nested_pb.lua
-- "Am looking for working example for below case where data is serialized as bytes to another protobuf."
-- https://ask.wireshark.org/question/33954/lua-example-for-protobuf-dissector-which-has-protobuf-encoded-within-protobuf/

-- Step 1 - document as you go. See header above and set_plugin_info().
local nested_pb_info =
{
    version = "1.0.0",
    author = "Good Coder",
    description = "Decode nested protobufs",
    repository = "Floppy in top drawer"
}

set_plugin_info(nested_pb_info)

do
    local protobuf_field_table = DissectorTable.get("protobuf_field") 
    local protobuf_dissector = Dissector.get("protobuf")

    protobuf_message_name_f = Field.new("protobuf.message.name")

    encoded_pb_p = Proto("encoded_pb_dissector", "Encoded Protobuf")
    encoded_pb_p.dissector = function(tvb, pinfo, subtree)

        -- What is the message type for the nested protobufs
        local message_type = {
            ["export_data.streamDataArgs"] = "message,encoded_data.encodeDataInfo",
            ["encoded_data.encodeDataInfo"] = "message,mac_data.macDataInfo"
        }

        -- Prevent recursion by only calling on last message name in tree
        local message_name = {
            ["message,encoded_data.encodeDataInfo"] = "encoded_data.encodeDataInfo",
            ["message,mac_data.macDataInfo"] = "mac_data.macDataInfo"
        }

        finfo_messages = { protobuf_message_name_f() }

        if (#finfo_messages > 0) then
            local call_dissector

            -- First time we're called there will be no pb_msg_type ...
(more)
edit flag offensive delete link more

Comments

Thanks Chuck, for providing the code. It works.

Followup question. if there is an example to parse repeated field from protobuf can you please point me the code to play with.

Below is more complicated encodeData.proto which includes the bytes data in a table.

syntax = "proto3";

package encoded_data;

message encodeData {
    encodeTable table;
}

message encodeTable {
    repeated encodeDataInfo row;
}

message encodeDataInfo { 
    string name = 1; // set to macData.proto
    bytes val = 2;     // serialized with macDataInfo from macData.proto
}
Babuvel gravatar imageBabuvel ( 2024-03-16 05:07:17 +0000 )edit

If you can provide a sample capture, I'll look at updating the code to handle this.

Chuckc gravatar imageChuckc ( 2024-03-16 11:07:37 +0000 )edit

Please find the details below.

encodeData.proto

syntax = "proto3";

package encoded_data;

message encodeData {
    encodeTable table = 1;
}

message encodeTable {
    repeated encodeDataInfo row = 1;
}

message encodeDataInfo { 
    string name = 1; 
    bytes val = 2; 
}

pcap file : https://www.dropbox.com/scl/fi/euta69...

Babuvel gravatar imageBabuvel ( 2024-03-17 05:22:00 +0000 )edit

I don't see a packet with two strings which would indicate multiple rows in the protobuf table?

Chuckc gravatar imageChuckc ( 2024-03-17 12:42:40 +0000 )edit

Please check the pcap file with table having multiple rows below.

row1: name: macData.proto, val: macDataInfo from macData.proto

row2: name: macData.proto, val: macDataInfo from macData.proto

row3: name: ipData.proto, val: ipDataInfo from ipData.proto

provided the ipData.proto info below for reference.

ipData.proto

syntax = "proto3";

package ip_data;

message ipDataInfo { 
    string ip_addr = 1; 
    uint64 time = 2; 
}

pcap: https://www.dropbox.com/scl/fi/mh8hki...

Babuvel gravatar imageBabuvel ( 2024-03-18 03:50:40 +0000 )edit
0

answered 2024-03-13 20:09:42 +0000

Chuckc gravatar image

-- DRAFT --
If this is the desired results, I'll clean up the lua code and post it along with config steps.

Protocol Buffers: /export_data.streamDataOut/streamData,request
    Message: export_data.streamDataArgs
        [Message Name: export_data.streamDataArgs]
        Field(1): Id = 1234 (int64)
            [Field Name: Id]
            [Field Type: int64 (3)]
            .000 1... = Field Number: 1
            .... .000 = Wire Type: varint (0)
            Value: d209
                Int64: 1234
            Id: 1234
        Field(2): data1  (bytes)
            [Field Name: data1]
            [Field Type: bytes (12)]
            .001 0... = Field Number: 2
            .... .010 = Wire Type: Length-delimited (2)
            Value Length: 42
            Value: 0a0d6d6163446174612e70726f746f12190a1130303a30303a30303a30303a31323a464610eb86b5af06
            data1: (42 bytes)
                All types
                    Protocol Buffers: encoded_data.encodeDataInfo
                        Message: encoded_data.encodeDataInfo
                            [Message Name: encoded_data.encodeDataInfo]
                            Field(1): name = macData.proto (string)
                                [Field Name: name]
                                [Field Type: string (9)]
                                .000 1... = Field Number: 1
                                .... .010 = Wire Type: Length-delimited (2)
                                Value Length: 13
                                Value: 6d6163446174612e70726f746f
                                    String: macData.proto
                                name: macData.proto
                            Field(2): val  (bytes)
                                [Field Name: val]
                                [Field Type: bytes (12)]
                                .001 0... = Field Number: 2
                                .... .010 = Wire Type: Length-delimited (2)
                                Value Length: 25
                                Value: 0a1130303a30303a30303a30303a31323a464610eb86b5af06
                                val: (25 bytes)
                                    All types
                                        Protocol Buffers: mac_data.macDataInfo
                                            Message: mac_data.macDataInfo
                                                [Message Name: mac_data.macDataInfo]
                                                Field(1): mac_addr = 00:00:00:00:12:FF (string)
                                                    [Field Name: mac_addr]
                                                    [Field Type: string (9)]
                                                    .000 1... = Field Number: 1
                                                    .... .010 = Wire Type: Length-delimited (2)
                                                    Value Length: 17
                                                    Value: 30303a30303a30303a30303a31323a4646
                                                        String: 00:00:00:00:12:FF
                                                    mac_addr: 00:00:00:00:12:FF
                                                Field(2): time = 1710048107 (uint64)
                                                    [Field Name: time]
                                                    [Field Type: uint64 (4)]
                                                    .001 0... = Field Number: 2
                                                    .... .000 = Wire Type: varint (0)
                                                    Value: eb86b5af06
                                                        Uint64: 1710048107
                                                    time: 1710048107

edit flag offensive delete link more

Comments

yes, this looks good.

Babuvel gravatar imageBabuvel ( 2024-03-13 21:20:36 +0000 )edit

Your Answer

Please start posting anonymously - your entry will be published after you log in or create a new account.

Add Answer

Question Tools

1 follower

Stats

Asked: 2024-03-09 07:50:49 +0000

Seen: 103 times

Last updated: Mar 24