Hello,
I find multiple duplicate fields in my JSON file that I get when I use tshark to convert pcap to JSON.
I know that this question has already been asked before, but the issue still persist:
https://ask.wireshark.org/question/505/deduplication-in-tshark-t-ek/ https://bugs.wireshark.org/bugzilla/show_bug.cgi?id=15719
I'm running tshark on windows 10 My tshark is of version:
tshark -version
TShark (Wireshark) 3.2.1 (v3.2.1-0-gbf38a67724d0)
I'm using the command:
tshark -r capture.pcap -T ek > packets.json
To generate my JSON file, it can be found here: https://www.elastic.co/blog/analyzing-network-packets-with-wireshark-elasticsearch-and-kibana
But when I try to push the JSON file to elasticsearch I get duplicate field error: Command:
curl -s -H "Content-Type: application/x-ndjson" -XPOST localhost:9200/test/_bulk --data-binary "@test.json" | jq
Error:
{ "took": 3, "errors": true, "items": [
{
"index": {
"_index": "packets-2020-04-07",
"_type": "doc",
"_id": "4YrWWHEBv6GDe8EVEwkp",
"status": 400,
"error": {
"type": "mapper_parsing_exception",
"reason": "failed to parse",
"caused_by": {
"type": "json_parse_exception",
"reason": "Duplicate field 'eth_eth_addr'\n at [Source: org.elasticsearch.common.bytes.AbstractBytesReference$MarkSupportingStreamInputWrapper@587b250c; line: 1, column: 1150]"
}
}
}
},
{
"index": {
"_index": "packets-2020-04-07",
"_type": "doc",
"_id": "4orWWHEBv6GDe8EVEwkp",
"status": 400,
"error": {
"type": "mapper_parsing_exception",
"reason": "failed to parse",
"caused_by": {
"type": "json_parse_exception",
"reason": "Duplicate field 'eth_eth_addr'\n at [Source: org.elasticsearch.common.bytes.AbstractBytesReference$MarkSupportingStreamInputWrapper@4f25e307; line: 1, column: 1130]"
}
}
}
}
]
}
I'm in the need of being able to handle fairly large files as fast as possible to quickly detect and find errors. So the idea of trying to write a program that goes through every field to check if it is a duplicate of an earlier field is not an realist solution for me.
