tshark export to ek json too slow?

asked 2018-07-28 08:34:28 +0000

This post is a wiki. Anyone with karma >750 is welcome to improve it.

I'm using this version of tshark

TShark (Wireshark) 2.4.5 (Git v2.4.5 packaged as 2.4.5-1)

running on Ubuntu Server 18.04 LTS, and the export to ek json (-T ek > file.json) is being too slow. Here an example:

alvaro@alvaro-server:~/PF_RING/userland/examples_zc$ time tshark -r file.pcap -T ek > file2.json 
^C

real    18m33.665s
user    14m2.707s
sys 4m3.751s

It didn't even finish and took 18 min...

edit retag flag offensive close merge delete

Comments

What's the size of the input file, use capinfos to show the details? Have you tried turning off name resolution -n? Can you share the capture file somewhere publicly?

grahamb ( 2018-07-28 09:50:21 +0000 )edit

These are the details of the pcap file:

alvaro@alvaro-server:~/PF_RING/userland/examples_zc$ capinfos file.pcap 
File name:           file.pcap
File type:           Wireshark/tcpdump/... - pcap
File encapsulation:  Ethernet
File timestamp precision:  microseconds (6)
Packet size limit:   file hdr: 262144 bytes
Number of packets:   33 M
File size:           2577 MB
Data size:           2035 MB
Capture duration:    29.995985 seconds
First packet time:   2018-07-28 09:17:49.004955
Last packet time:    2018-07-28 09:18:19.000940
Data byte rate:      67 MBps
Data bit rate:       542 Mbps
Average packet size: 60.00 bytes
Average packet rate: 1130 kpackets/s
SHA1:                5b0fd55620b8a3759362b68a3e88e3ec476ea634
RIPEMD160:           ce86d8074cb92fee3a66d6200cb5bdeebd7ab3d6
MD5:                 5806f2cfad7fefcdb1065ed2991be326
Strict time order:   True
Number of interfaces in file: 1
Interface #0 info:
                     Encapsulation = Ethernet (1 - ether)
                     Capture length = 262144
                     Time precision = microseconds (6)
                     Time ticks per second = 1000000
                     Number of stat entries = 0
                     Number of packets = 33916811

I tried now turning off name resolution, but it doesn ...(more)

alvarospunk ( 2018-07-28 10:03:40 +0000 )edit

file uploaded

alvarospunk ( 2018-07-28 11:35:46 +0000 )edit

add a comment

answered 2018-07-28 18:28:00 +0000

grahamb

flag of United Kingdom of Great Britain and Northern Ireland

23790 ●4 ●950 ●227 https://www.wireshark.org

I think you're just hitting the limits of working with large capture files.

The file has 33M packets. Splitting the file into 1M packets chunks using editcap -c 1000000 file.pcap out.pcapng and then running the same tshark output on a 1M file took 144 seconds (i7-8550U with SSD), the tshark process peaked at 2GB RAM and the output file was 3.88 GB.

edit flag offensive delete link

Comments

Thank you for your answer. Unfortunately, I think you're right and makes sense (I have to deal with larger pcap files and will try to find another solution).

alvarospunk ( 2018-07-28 21:45:03 +0000 )edit

add a comment

tshark export to ek json too slow?

Comments

1 Answer

Comments

Your Answer

Question Tools

Stats

Related questions

tshark export to ek json too slow? edit

Comments

1 Answer

Comments

Your Answer

Question Tools

Stats

Related questions

tshark export to ek json too slow?