Ask Your Question
0

Merge regular text logs (as info) and packet captures

asked 2020-06-20 18:39:15 +0000

le_top gravatar image

I have regular human readable text logs from my application showing when certain activity was done. They do not have any information that should be presented as a packet.

A I packet logs with the low level packets that are decoded with wireshark.

I would like to read them both in wireshark so that the regular log lines are interleaved with the packet logs. The information from the regular log file could be shown in the "info" colkumn.

I see three potential "methods":

  • The log file has timestamps. So these could be used to interleave the information fairly properly.
  • I could also pipe the text log to a converter that would take the incoming text lines and convert them to a pcap file.
  • Ultimately I could also generate a pcap file directly from the application as wireshark can already read multiple logs or they can be merged using existing tools.

I made some searches, but I could not find a hint about any working solution that would be able to achieve my goal using one of these methods.

The purpose is to avoid having multiple tools open and match the timestamps with what I see in the text viewer(s) and the wireshark log - it would be nice to have it all in a single thread.

edit retag flag offensive close merge delete

3 Answers

Sort by ยป oldest newest most voted
0

answered 2020-06-21 01:57:23 +0000

le_top gravatar image

updated 2020-06-21 02:16:40 +0000

I created a perl script to convert the log file to a pcap file with syslog messages. Available here: https://gist.github.com/mdeweerd/47ea... .

After conversion, I just need to drag & drop the pcap files to wireshark to see them all.

Version at the time of this post - I'll probably add the possibility to apply a timeshift:

#!/bin/perl
# SCRIPT TO CONVERT TEXT LOG TO PCAP FILE
# READS DATES FROM LOG FILE
# WHEN READING FROM PIPE (UNTESTED) WITHOUT DATES,
# THE CURRENT TIMESTAMP IS USED.
#
# 0, 1 or 2 parameters to the script:
#  - When 0 parameters: uses STDIN, and writes to file.pcap.
#  - When 1 parameters: Depends on extension: either .pcap file or log file.
#  - When 2 parameters: One of the files should have the .pcap extension
use File::PCAP::Writer;
use Time::HiRes qw(gettimeofday);
use Time::Piece;
use Date::Parse;
use Socket;

use List::Util qw(sum);
use POSIX qw(locale_h);
use locale;

# Make sure text is in English.
setlocale(LC_CTYPE);

my $fname='file.pcap'; # Default output file name
my $logname='';        # Default log file, when '', use STDIN

# Check CLI paramaters
if(scalar(@ARGV)==2) {
   if( $ARGV[0] =~ /.pcap$/ ) {
      $fname=$ARGV[0];
      $logname=$ARGV[1];
   } elsif( $ARGV[1] =~ /.pcap$/ ) {
      $fname=$ARGV[1];
      $logname=$ARGV[0];
   }

} elsif(scalar(@ARGV)==1) {
   if( $ARGV[0] =~ /.pcap$/ ) {
      $fname=$ARGV[0];
   } else {
      $logname=$ARGV[0];
   }
}

# Delete the output file
unlink($fname);
# Open the output file
my $fpw = File::PCAP::Writer->new( {
    fname => $fname,
    dlt => 1,
} );

# Open the input file
my $fh;
my $timeFromFile=0;
if($logname ne '') {
   open( $fh => $logname) || die "Cannot open $logname: $!";
   $timeFromFile=1;
} else {
   $fh=*STDIN;
   $logname="STDIN";
}

# Inform about what we are doing
print "Convert $logname to $fname";

# Do conversion
my $t=0;
my $usec;
while( my $line = <$fh> ) {
    chomp;
    if(length($line)==0||$line=~/^\s*$/) {
        next;
    }


    #print $t;
    if($line=~/^\[([^[]*?)\]\s?(.*)/) {
       my $timestr=$1;
       $line=$2;
       $t=str2time($1,"+0000");
       #print $timestr." ".$t;
       $usec = 0;
       $timeFromFile=1;
    } elsif(!$timeFromFile) {
       ($t, $usec) = gettimeofday();
    }
    $line="<134>".$line;
    #$line=gmtime($t)->strftime("%M %b %H:%M:%S")." ".$line;
    #print $line."\n";
    $buf=getPacketForText($line);
    $blen=$plen=length($buf);
    $fpw->packet( 0+$t, $usec, $blen, $plen, $buf );
}
#$fpw->close();


# Helper functions, adapted from  https://stackoverflow.com/questions/51061864/sending-spoofed-syslog-messages-in-perl


sub udp_checksum {
    # thanks to ikegami in post https://stackoverflow.com/questions/46181281/udp-checksum-function-in-perl
    my $packed_src_addr = shift;  # As 4-char string, e.g. as returned by inet_aton.
     my $packed_dst_addr = shift;  # As 4-char string, e.g. as returned by inet_aton.
    my $udp_packet      = shift;  # UDP header and data as a string.

    my $sum = sum(
    IPPROTO_UDP,
    length($udp_packet),
    map({ unpack('n*', $_) }
        $packed_src_addr,
        $packed_dst_addr,
        $udp_packet."\0",  # Extra byte ignored if length($udp_packet) is even.
        ),
    );

    while (my $hi = $sum >> 16) {
        $sum = ($sum & 0xFFFF) + $hi;
    }

    return ~$sum & 0xFFFF;
}

sub getPacketForText {
    $data=shift;

    $src_host = "localhost";
    $dst_host = 127.0.0.1;
    $src_port = 33333;
    $dest_port = 514;
    $cksum = 0; #initialise, we will sort this later

    #$udp_len is the udp packet length in the udp header. Its just 8 plus the length of the data
    #for this test harness we ...
(more)
edit flag offensive delete link more
0

answered 2020-06-20 22:21:04 +0000

Chuckc gravatar image

updated 2020-06-20 22:22:17 +0000

I was kinda hoping Guy would weave Fileshark into his answer. Oh well.
Here's an old email thread that might help.
Until then, how about turning the logs into syslog packets, tweak the timestamps then merge with the "low level" packets?

edit flag offensive delete link more

Comments

So that'd be "syslog over UDP over IP over (whatever the link layer for the capture is)".

Guy Harris gravatar imageGuy Harris ( 2020-06-21 00:48:25 +0000 )edit

I check the email thread, but it is IMHO about extracting files from packets. Converting in syslog packets is more or less like the method to convert the data into "unused" ethernet packets. I made a quick and dirty perl script to do it.

le_top gravatar imagele_top ( 2020-06-21 01:59:08 +0000 )edit
0

answered 2020-06-20 19:45:32 +0000

Guy Harris gravatar image

The log file has timestamps. So these could be used to interleave the information fairly properly.

Intereaving has to be done in a file, which requires a file format that supports text logs.

I could also pipe the text log to a converter that would take the incoming text lines and convert them to a pcap file.

pcap format is a format in which all records are packet records, with the packets all having a particular link-layer header type. In order to put textual message sin a pcap file, you would have to "fake it" by, for example, using a currently-unused Ethernet type to mean "this isn't a packet, it's a log message", add a dissector for that Ethernet type, and hope that all your captures will have link-layer headers that include an Ethernet type.

pcapng format, however, is more flexible; for example, a Text Log Message Block could be added, with a time stamp, options, and a body that just contains UTF-8 text. Either the time stamp would have to have a resolution hardwired for all such blocks or there would have to be blocks indicating message sources. similar to Interface Description Blocks. I'd be inclined to vote for the latter.

Proposals for new block types are probably best done as pull requests for the pcapng specification repository.

edit flag offensive delete link more

Comments

Thank you for your feedback. Apparently there is not method ready to use. Adding stuff to the pcapng format is the good way to go for the long term, to make a proposal I seemingly have to dig into how the specification repository works.

le_top gravatar imagele_top ( 2020-06-20 23:05:23 +0000 )edit

Your Answer

Please start posting anonymously - your entry will be published after you log in or create a new account.

Add Answer

Question Tools

1 follower

Stats

Asked: 2020-06-20 18:39:15 +0000

Seen: 52 times

Last updated: Jun 21