Ask Your Question

Revision history [back]

duplicate fields -T ek

Hello,

I find multiple duplicate fields in my JSON file that I get when I use tshark to convert pcap to JSON.

I know that this question has already been asked before, but the issue still persist:

https://ask.wireshark.org/question/505/deduplication-in-tshark-t-ek/ https://bugs.wireshark.org/bugzilla/show_bug.cgi?id=15719

I'm running tshark on windows 10 My tshark is of version:

tshark -version
TShark (Wireshark) 3.2.1 (v3.2.1-0-gbf38a67724d0)

I'm using the command:

tshark -r capture.pcap -T ek > packets.json

To generate my JSON file, it can be found here: https://www.elastic.co/blog/analyzing-network-packets-with-wireshark-elasticsearch-and-kibana

But when I try to push the JSON file to elasticsearch I get duplicate field error: Command:

curl -s -H "Content-Type: application/x-ndjson" -XPOST localhost:9200/test/_bulk --data-binary "@test.json" | jq

Error:

{ "took": 3, "errors": true, "items": [ { "index": { "_index": "packets-2020-04-07", "_type": "doc", "_id": "4YrWWHEBv6GDe8EVEwkp", "status": 400, "error": { "type": "mapper_parsing_exception", "reason": "failed to parse", "caused_by": { "type": "json_parse_exception", "reason": "Duplicate field 'eth_eth_addr'\n at [Source: org.elasticsearch.commo[email protected]587b250c; line: 1, column: 1150]" } } } }, { "index": { "_index": "packets-2020-04-07", "_type": "doc", "_id": "4orWWHEBv6GDe8EVEwkp", "status": 400, "error": { "type": "mapper_parsing_exception", "reason": "failed to parse", "caused_by": { "type": "json_parse_exception", "reason": "Duplicate field 'eth_eth_addr'\n at [Source: org.elasticsearch.common.bytes.AbstractBytesReference$M[email protected]; line: 1, column: 1130]" } } } }
] }

I'm in the need of being able to handle fairly large files as fast as possible to quickly detect and find errors. So the idea of trying to write a program that goes through every field to check if it is a duplicate of an earlier field is not an realist solution for me.

duplicate fields -T ek

Hello,

I find multiple duplicate fields in my JSON file that I get when I use tshark to convert pcap to JSON.

I know that this question has already been asked before, but the issue still persist:

https://ask.wireshark.org/question/505/deduplication-in-tshark-t-ek/ https://bugs.wireshark.org/bugzilla/show_bug.cgi?id=15719

I'm running tshark on windows 10 My tshark is of version:

tshark -version
TShark (Wireshark) 3.2.1 (v3.2.1-0-gbf38a67724d0)

I'm using the command:

tshark -r capture.pcap -T ek > packets.json

To generate my JSON file, it can be found here: https://www.elastic.co/blog/analyzing-network-packets-with-wireshark-elasticsearch-and-kibana

But when I try to push the JSON file to elasticsearch I get duplicate field error: Command:

curl -s -H "Content-Type: application/x-ndjson" -XPOST localhost:9200/test/_bulk --data-binary "@test.json" | jq

Error:

{   "took": 3,   "errors": true,   "items": [
    {
      "index": {
        "_index": "packets-2020-04-07",
        "_type": "doc",
        "_id": "4YrWWHEBv6GDe8EVEwkp",
        "status": 400,
        "error": {
          "type": "mapper_parsing_exception",
          "reason": "failed to parse",
          "caused_by": {
            "type": "json_parse_exception",
            "reason": "Duplicate field 'eth_eth_addr'\n at [Source: org.elasticsearch.commo[email protected]587b250c; line: 1, column: 1150]"
          }
        }
      }
    },
    {
      "index": {
        "_index": "packets-2020-04-07",
        "_type": "doc",
        "_id": "4orWWHEBv6GDe8EVEwkp",
        "status": 400,
        "error": {
          "type": "mapper_parsing_exception",
          "reason": "failed to parse",
          "caused_by": {
            "type": "json_parse_exception",
            "reason": "Duplicate field 'eth_eth_addr'\n at [Source: org.elasticsearch.commo[email protected]4f25e307; line: 1, column: 1130]"
          }
        }
      }
    } 
] }

}

I'm in the need of being able to handle fairly large files as fast as possible to quickly detect and find errors. So the idea of trying to write a program that goes through every field to check if it is a duplicate of an earlier field is not an realist solution for me. me.