Ask Your Question

calculate md5 for each packet and output to new file with updated field

asked 2020-06-24 16:51:46 +0000

DJKR gravatar image

updated 2020-06-24 16:52:44 +0000


I have a file with no md5 checksum, I want to calculate the md5 checksum for each packet and output to a new pcap file.

I saw this thread

However not sure if the command was cut short, but it gives me an error

tshark -o frame.generate_md5_hash:TRUE -w pcap_sig_hem_2020_06_24_14_53.md5.pcap -r pcap_sig_hem_2020_06_24_14_53.pcap $(tshark -o frame.generate_md5_hash:TRUE -r pcap_sig_hem_2020_06_24_14_53.pcap -T fields -e frame.md5_hash)

Running as user "root" and group "root". This could be dangerous.

tshark: "7f0ec68a82b5b2e86c7642477cd4b7e3" was unexpected in this context.

Thanks for any help in advance.

edit retag flag offensive close merge delete


Can you explain a bit of what the end goal is? File integrity? Duplicate detection?

Chuckc gravatar imageChuckc ( 2020-06-24 17:07:01 +0000 )edit

Duplicate detection.

DJKR gravatar imageDJKR ( 2020-06-24 17:56:59 +0000 )edit

OK, so do you want a list of duplicates or a capture file of duplicates?

grahamb gravatar imagegrahamb ( 2020-06-24 18:08:09 +0000 )edit

A list of duplicate packets in the file would be fine as long as I can go back to original capture file and using the list produced filter on those packets or a capture file of duplicates.

DJKR gravatar imageDJKR ( 2020-06-24 18:12:53 +0000 )edit

1 Answer

Sort by ยป oldest newest most voted

answered 2020-06-24 18:12:40 +0000

grahamb gravatar image

updated 2020-06-24 18:16:52 +0000

Looking at the linked email, yes the command is snipped as the full command is shown earlier in the text, the last command simply ensures the preference to generate an MD5 hash is enabled. The full command is:

tshark -o frame.generate_md5_hash:TRUE \
  $(tshark -r MYFILE.PCAP -Tfields -e frame.md5_hash \
  | sort \
  | uniq -c \
  | sort -n -r \
  | grep -v ' 1 ' \
  | awk 'BEGIN {printf "frame.number==0"} \
    {printf "||frame.md5_hash=="$2} END {print ""}')
edit flag offensive delete link more


Tested on WSL:

$ cat ./tshark_dupes

tshark.exe -o frame.generate_md5_hash:TRUE \
  $(tshark.exe -o frame.generate_md5_hash:TRUE -r MYFILE.PCAP -Tfields -e frame.md5_hash \
  | sort \
  | uniq -c \
  | sort -n -r \
  | grep -v ' 1 ' \
  | tr -d '\r' \
  | awk 'BEGIN { printf "frame.number==0" } \
    { printf " || frame.md5_hash==%s",$2 } END { print "" }')
Chuckc gravatar imageChuckc ( 2020-06-24 19:14:16 +0000 )edit

(This is more of a comment to go along with @grahamb answer but still haven't figured out how to add screenshots to comments)
MD5 hashes added in this commit
It is a generated field created if the preference frame.generate_md5_hash is set to TRUE.

if (generate_md5_hash) {
ti = proto_tree_add_string(fh_tree, hf_frame_md5_hash, tvb, 0, 0, digest_string);

proto_item_set_generated is used to mark fields as not being read from the
captured data directly, but inferred from one or more values.

Simple example with with hashes copied to GUI display filter for testing:

$ tshark -o frame.generate_md5_hash:TRUE -r ./output.pcap -T fields -e frame.md5_hash | sort | uniq -c | sort -n | tail -5
      1 fd04a0e73be79cc94019a68a00666920
      1 feff0464f7b76af44a8d4acfc84bfafe
      3 42d98bc2aa77e47d09185cb8d98a0f9e
      3 a2e5d2e10a18a6bb30f6712088b7a293
      5 e29a39f345d4099d3205c9355a004bab

Chuckc gravatar imageChuckc ( 2020-06-24 19:38:56 +0000 )edit

So I have tested this on Centos 7and WSL and find that the output file generated seems to be missing the MD5 hash

I can see the hash for each packet is being generated

Apologies about the formatting below

tshark -o frame.generate_md5_hash:TRUE -r ./co_sip_signalling_2253_24062020.pcap -T fields -e frame.md5_hash | sort | uniq -c | sort -n | tail -5
Running as user "root" and group "root". This could be dangerous.
 12 bb0d6f72a79dd2424d1b39242262b4c4
 12 dc475ce26962e449218a5836dee81712
 12 dd170db01393915780f41cfac6a9e59c
 12 eb09ed23bd8bba1e9e9eb82edef30be5
 12 f3aa183d899399c65a00170ee53cddef

If I run the second part of the command I can see the field name with the md5 hash being generated

tshark -o frame.generate_md5_hash:TRUE -r co_sip_signalling_2253_24062020.pcap -Tfields -e frame.md5_hash \
>   | sort \
>   | uniq -c \
>   | sort -n -r \
>   | grep -v ' 1 ' \
>   | tr -d '\r' \
>   | awk 'BEGIN { printf "frame.number==0" } \
>     { printf " || frame.md5_hash==%s",$2 } END { print "" }'
frame.number==0Running as user ...
DJKR gravatar imageDJKR ( 2020-06-24 22:14:48 +0000 )edit

"the output file generated seems to be missing the MD5 hash"
That is correct. It is a "generated" field created from other data in the capture file.
It's ephemeral - only available if the preference is set to create it then gone at the end of the session.

If you plan to make use of it often then set the preference to TRUE and the hashes will be created and available for all files you open.

Chuckc gravatar imageChuckc ( 2020-06-24 23:36:35 +0000 )edit

Formatting hints:

Highlight required text and use Code button or Ctrl + K, or whatever shows up in the tool tip for the Code button to format as code, or indent by 4 spaces or use <pre></pre> tags.

Use <img src="path/to/image" width="640" /> to display images and constrain them to a sensible width.

Use <br/><br/> to put a blank line into a comment.

grahamb gravatar imagegrahamb ( 2020-06-25 07:44:31 +0000 )edit

Your Answer

Please start posting anonymously - your entry will be published after you log in or create a new account.

Add Answer

Question Tools

1 follower


Asked: 2020-06-24 16:51:46 +0000

Seen: 47 times

Last updated: Jun 24