Ask Your Question
0

Converting Pcap file to CSV file while defautly keeping all features/fields defined in pcap

asked 2020-05-12 15:03:58 +0000

Artemis gravatar image

Hi all, I have been searching for solutions that can transform PCAP files to CSV format while keeping all the details defined in the pcap file, without explicitly defining which features/fields that i want to include in CSV format.

With Wireshark&Tshark, i have tested the following two ways:

  1. the first one is to use tshark commands such as:

    tshark -r traffic.pcap > traffic.csv

    tshark -r traffic.pcap -T fields -e ip.src -E separator=, -E occurrence=f > traffic.csv

    In any case, the traffic.csv contains only general information (e.g., No., Time, Source, Destination, Length, Protocol, Info) of the traces without any packet detail(Packet Bytes).

  2. I also tried with Wireshark by selecting Export Packet Dissections, however i got differet results by exporting the same pcap file to csv, json, and plain text format. In general, the exported csv file still contains only general information of packets, without packet detail (Packet Bytes), even i selected the field (Packet Bytes during export).

I wonder if there is a way i can also have the the details of each packet in the csv file? Thanks.

edit retag flag offensive close merge delete

Comments

What would you expect a CSV of all fields to look like? A CSV file is a representation of data in a row\column format, with the rows as packets and the columns as fields. Depending on the data the number of columns could be enormous.

Maybe you could explain what you want to do with the data so alternatives could be suggested?

grahamb gravatar imagegrahamb ( 2020-05-12 15:54:13 +0000 )edit

the dimension of the data is not an issue, since i would like to explore the different column combinations to support the machine learning work later. One thing i found wierd is that, if I choose the json or plain text as an export format, the packet byte is in the file, it does not work with csv file.

Artemis gravatar imageArtemis ( 2020-05-12 15:59:37 +0000 )edit

CSV needs to know what the columns are in order to put the value into the correct columns. Maybe look at the ElasticSearch output, -T ek.

grahamb gravatar imagegrahamb ( 2020-05-12 16:19:27 +0000 )edit

I don't know if i'm being naive thinking that the automatic conversion can leave fields as empty if certain features are not appicable to specific packets, I will check the ElasticSearch as suggested thanks.

Artemis gravatar imageArtemis ( 2020-05-12 19:22:00 +0000 )edit

1 Answer

Sort by ยป oldest newest most voted
0

answered 2020-05-12 19:19:37 +0000

Guy Harris gravatar image

It's not clear that the concept of a CSV of all fields is well-defined.

A row of a CSV file is just a Comma-Separated sequence of Values; there are no tags to indicate what the values are values of. A CSV row, containing the values of all the fields in a packet, would just be a sequence of values, with no indication what those values signify - and not all rows would have the same number of values.

If you used the first row as a table of field names, to solve that problem, that would require that the first row have the name of each field that appears in a packet in the file, and that elements in subsequent rows may be empty (if the fed in question isn't in the packet corresponding to that row). It might also require either that a field name may appear more than once in the first row, to handle packets with more than one instance of a field.

There is currently no code in Wireshark to do that.

One thing i found wierd is that, if I choose the json or plain text as an export format, the packet byte is in the file, it does not work with csv file.

What do you mean by "the packet byte(s)"? What's exported as JSON are the packet fields; what's shown is the value, in a somewhat human-readable form, which isn't necessarily the raw bytes.

What's exported as CSV are the columns; the columns have the advantage that there is a fixed set of columns specified, so that you don't have the problems I mentioned above with CSV dumping of packet fields.

I.e., there is a reason why CSV export is different from JSON or PDML exports.

For plain text exports, which are intended for humans to read, you can choose to export the columns as displayed in the packet list pane, the packet details as displayed in the packet details pane, or the raw hex/characters dump as displayed in the hex/characters dump pane; more than one of those can be chosen.

edit flag offensive delete link more

Comments

Thanks for the kind isnght, I understand better now the first point. Regarding the second one "it does not work with csv file", i mean is it normal that CSV does not contain any information from packet details pane? i understand the json and plain text formats only structure the packet details in a different way, is csv file supposed to contain all fileds from packet detail panes (the packet byte such as frame attrbutes, eth attrbutes), the situation is that i only got attributes as (No., Time, Source, Destination, Length, Protocol, Info) in csv format, even if i selected the "packet byte" option during export.

Artemis gravatar imageArtemis ( 2020-05-12 19:42:28 +0000 )edit

is it normal that CSV does not contain any information from packet details pane?

Yes.

i understand the json and plain text formats only structure the packet details in a different way,

The JSON format is packet details; the plain text format can be packet summaries, packet details, raw packet data, or any combination thereof.

is csv file supposed to contain all fileds from packet detail panes (the packet byte such as frame attrbutes, eth attrbutes),

No.

the situation is that i only got attributes as (No., Time, Source, Destination, Length, Protocol, Info) in csv format, even if i selected the "packet byte" option during export.

The only way you can select any of those during export to CSV is to choose another format that allows you to select them, and then change the format to CSV. If the format chosen is CSV, those options are greyed-out to indicate that ...(more)

Guy Harris gravatar imageGuy Harris ( 2020-05-12 23:48:23 +0000 )edit

Thanks a lot for the clarification.

Artemis gravatar imageArtemis ( 2020-05-13 07:55:03 +0000 )edit

Your Answer

Please start posting anonymously - your entry will be published after you log in or create a new account.

Add Answer

Question Tools

1 follower

Stats

Asked: 2020-05-12 15:03:58 +0000

Seen: 632 times

Last updated: May 12