tshark crashes: reading large packet captures via lua_script

asked 2019-02-18 12:13:19 +0000

Dharun
1 ●1 ●1

Hi, I am trying to read and process a packet captures(pcap)of 1 GB, using tshark & lua_script using listerners, but the system memory gradually ramps up and tshark crashes when the system memory is maxed out..

Code flow is based on : http://wiretrail.guru/kager/lua-lesso...

Any inputs on how to handle large files this ?, Even observed this crash for 400MB+ files as well.
My understanding is Listener is holding the memory of each packet until we complete the full packet capture , Is there a way to free up memory after processing every packet ?

example: tshark -q -r test.pcap -X lua_script:process.lua -X lua_script1:test.pcap

Is it possible to read the packets in batches rather than loading full pcap file

Regards Dharun

answered 2019-02-18 14:38:25 +0000

Anders

5023 ●97 ●58

updated 2019-02-19 16:11:22 +0000

cmaynard

11115 ●11 ●317 ●165 https://www.igt.com/

In general tshark/wireshark will consume more and more memory as more packets are read in. The only workaround is to split the file in manageable chunks. Editcap is one tool that can be used for this purpose. That said, I haven't had problems handling files > 500 MB (other than the time it takes to filter them), so possibly there is something sub-optimal in your code.

edit flag offensive delete link

Comments

For more ideas, see https://wiki.wireshark.org/KnownBugs/...

cmaynard ( 2019-02-19 17:54:05 +0000 )edit

My comment to have more informations about the Listener/Fields memory management with tshark and command line.

When Wireshark profiles are correctly tuned (few protocols, disabled TCP reassembly & analysis, bytes tracking, timestamps calculations, IP defrag), tshark memory doesn't grow at all.

Simply adding a simple script with 1 listener and 1 field, without doing nothing with them, then the memory grows along with pcap file.

As tshark memory alone doesn't grow due to correct profile, I'd have expected Lua to forget about all values also on each Listener.packet() loop, but it doesn't seem to be coded like this.

Lua script by itself doesn't consume that much memory according to collectgarbage("count").

So it seems that Lua+Listener+Field is forcing tshark to collect all packets as if some reassembly was asked.

Can some developers give some hints about the section of code involved here ?

TomLaBaude ( 2020-10-05 15:09:42 +0000 )edit

Hi, I'm no user of LUA but in folder wslua/wslua_listener.c and epan/tap.c perhaps.

Anders ( 2020-10-06 07:45:49 +0000 )edit

add a comment

Your Answer

Please start posting anonymously - your entry will be published after you log in or create a new account.

tshark crashes: reading large packet captures via lua_script

1 Answer

Comments

Your Answer

Question Tools

Stats

Related questions

tshark crashes: reading large packet captures via lua_script edit

1 Answer

Comments

Your Answer

Question Tools

Stats

Related questions

tshark crashes: reading large packet captures via lua_script