Ask Your Question
0

tshark crashes: reading large packet captures via lua_script

asked 2019-02-18 12:13:19 +0000

Dharun gravatar image

Hi, I am trying to read and process a packet captures(pcap)of 1 GB, using tshark & lua_script using listerners, but the system memory gradually ramps up and tshark crashes when the system memory is maxed out..

Code flow is based on : http://wiretrail.guru/kager/lua-lesso...

  1. Any inputs on how to handle large files this ?, Even observed this crash for 400MB+ files as well.

  2. My understanding is Listener is holding the memory of each packet until we complete the full packet capture , Is there a way to free up memory after processing every packet ?

example: tshark -q -r test.pcap -X lua_script:process.lua -X lua_script1:test.pcap

  1. Is it possible to read the packets in batches rather than loading full pcap file

Regards Dharun

edit retag flag offensive close merge delete

1 Answer

Sort by ยป oldest newest most voted
0

answered 2019-02-18 14:38:25 +0000

Anders gravatar image

updated 2019-02-19 16:11:22 +0000

cmaynard gravatar image

In general tshark/wireshark will consume more and more memory as more packets are read in. The only workaround is to split the file in manageable chunks. Editcap is one tool that can be used for this purpose. That said, I haven't had problems handling files > 500 MB (other than the time it takes to filter them), so possibly there is something sub-optimal in your code.

edit flag offensive delete link more

Comments

cmaynard gravatar imagecmaynard ( 2019-02-19 17:54:05 +0000 )edit

My comment to have more informations about the Listener/Fields memory management with tshark and command line.

When Wireshark profiles are correctly tuned (few protocols, disabled TCP reassembly & analysis, bytes tracking, timestamps calculations, IP defrag), tshark memory doesn't grow at all.

Simply adding a simple script with 1 listener and 1 field, without doing nothing with them, then the memory grows along with pcap file.

As tshark memory alone doesn't grow due to correct profile, I'd have expected Lua to forget about all values also on each Listener.packet() loop, but it doesn't seem to be coded like this.

Lua script by itself doesn't consume that much memory according to collectgarbage("count").

So it seems that Lua+Listener+Field is forcing tshark to collect all packets as if some reassembly was asked.

Can some developers give some hints about the section of code involved here ?

TomLaBaude gravatar imageTomLaBaude ( 2020-10-05 15:09:42 +0000 )edit

Hi, I'm no user of LUA but in folder wslua/wslua_listener.c and epan/tap.c perhaps.

Anders gravatar imageAnders ( 2020-10-06 07:45:49 +0000 )edit

Your Answer

Please start posting anonymously - your entry will be published after you log in or create a new account.

Add Answer

Question Tools

1 follower

Stats

Asked: 2019-02-18 12:13:19 +0000

Seen: 231 times

Last updated: Feb 19 '19