Ask Your Question
0

tshark export to ek json too slow?

asked 2018-07-28 08:34:28 +0000

this post is marked as community wiki

This post is a wiki. Anyone with karma >750 is welcome to improve it.

I'm using this version of tshark

TShark (Wireshark) 2.4.5 (Git v2.4.5 packaged as 2.4.5-1)

running on Ubuntu Server 18.04 LTS, and the export to ek json (-T ek > file.json) is being too slow. Here an example:

[email protected]:~/PF_RING/userland/examples_zc$ time tshark -r file.pcap -T ek > file2.json 
^C

real    18m33.665s
user    14m2.707s
sys 4m3.751s

It didn't even finish and took 18 min...

edit retag flag offensive close merge delete

Comments

What's the size of the input file, use capinfos to show the details? Have you tried turning off name resolution -n? Can you share the capture file somewhere publicly?

grahamb gravatar imagegrahamb ( 2018-07-28 09:50:21 +0000 )edit

These are the details of the pcap file:

[email protected]:~/PF_RING/userland/examples_zc$ capinfos file.pcap 
File name:           file.pcap
File type:           Wireshark/tcpdump/... - pcap
File encapsulation:  Ethernet
File timestamp precision:  microseconds (6)
Packet size limit:   file hdr: 262144 bytes
Number of packets:   33 M
File size:           2577 MB
Data size:           2035 MB
Capture duration:    29.995985 seconds
First packet time:   2018-07-28 09:17:49.004955
Last packet time:    2018-07-28 09:18:19.000940
Data byte rate:      67 MBps
Data bit rate:       542 Mbps
Average packet size: 60.00 bytes
Average packet rate: 1130 kpackets/s
SHA1:                5b0fd55620b8a3759362b68a3e88e3ec476ea634
RIPEMD160:           ce86d8074cb92fee3a66d6200cb5bdeebd7ab3d6
MD5:                 5806f2cfad7fefcdb1065ed2991be326
Strict time order:   True
Number of interfaces in file: 1
Interface #0 info:
                     Encapsulation = Ethernet (1 - ether)
                     Capture length = 262144
                     Time precision = microseconds (6)
                     Time ticks per second = 1000000
                     Number of stat entries = 0
                     Number of packets = 33916811

I tried now turning off name resolution, but it doesn ...(more)

alvarospunk gravatar imagealvarospunk ( 2018-07-28 10:03:40 +0000 )edit

file uploaded

alvarospunk gravatar imagealvarospunk ( 2018-07-28 11:35:46 +0000 )edit

1 Answer

Sort by ยป oldest newest most voted
0

answered 2018-07-28 18:28:00 +0000

grahamb gravatar image

I think you're just hitting the limits of working with large capture files.

The file has 33M packets. Splitting the file into 1M packets chunks using editcap -c 1000000 file.pcap out.pcapng and then running the same tshark output on a 1M file took 144 seconds (i7-8550U with SSD), the tshark process peaked at 2GB RAM and the output file was 3.88 GB.

edit flag offensive delete link more

Comments

Thank you for your answer. Unfortunately, I think you're right and makes sense (I have to deal with larger pcap files and will try to find another solution).

alvarospunk gravatar imagealvarospunk ( 2018-07-28 21:45:03 +0000 )edit

Your Answer

Please start posting anonymously - your entry will be published after you log in or create a new account.

Add Answer

Question Tools

1 follower

Stats

Asked: 2018-07-28 08:34:28 +0000

Seen: 134 times

Last updated: Jul 28