Using Lua to tag SYN-ACK followed by a RST

edit

Apart from the (very flexible) Lua solution, you could also use the TCP completeness filter to achieve (kind of) the same. The filter tcp.completeness & 62 == 34will display all TCP sessions in which there was seen:

• a SYN/ACK (2)
• no bare ACK (4)
• no data (8)
• no FIN (16)
• a RST (32)

I'm leaving out the SYN(1) as there might be sessions where the capture started straight after the SYN, if you only want sessions in which the SYN is also present, you can just use tcp.completeness == 35

I didn't verify the logic of "SYN-ACKs which result in an RST packet" but this will load and execute clean.

-- Create a new protocol to hold the custom field.
-- This protocol will be used for tagging TCP packets in Wireshark.
local tcp_tagging_proto = Proto("tcp_tagging", "TCP Tagging Protocol")

-- Define a custom field for tagging purposes.
-- This field will be used to indicate whether a TCP SYN packet is followed by an RST packet.
local f_tcp_syn_followed_by_rst = ProtoField.bool("tcp_tagging.syn_followed_by_rst", "SYN followed by RST", 8, nil, 0x01)

tcp_tagging_proto.fields = { f_tcp_syn_followed_by_rst }

-- Tables to hold the stream information
-- This table will keep track of TCP streams that have an RST packet.
local rst_stream_table = {}

-- Define field extractors
local tcp_stream_field = Field.new("tcp.stream")
local tcp_flags_field = Field.new("tcp.flags")

-- Function to analyze RST packets and register them with their stream number.
-- This function will be called for each packet to check if it is an RST packet and record its stream number.
-- Function to analyze RST packets and register them with their stream number.
local function analyze_packet_rst(pinfo, tvb, tree)
    local tcp_stream = tcp_stream_field().value -- Extract the numeric value
    local flags = tcp_flags_field().value -- Extract the numeric value

    local rst_flag = bit32.band(flags, 0x04) ~= 0 -- Check if the RST flag is set.

    -- If the RST flag is set, record the stream number in the rst_stream_table.
    if rst_flag then
        rst_stream_table[tcp_stream] = true
    end
end

-- Function to find SYN-ACK packets and tag them if an RST was found for the same stream.
-- This function tags SYN-ACK packets that are followed by an RST packet in the same TCP stream.
local function analyze_packet_tag(pinfo, tvb, tree)
    local tcp_stream = tcp_stream_field().value -- Extract the numeric value
    local flags = tcp_flags_field().value -- Extract the numeric value

    local syn_flag = bit32.band(flags, 0x02) ~= 0 -- Check if the SYN flag is set.
    local ack_flag = bit32.band(flags, 0x10) ~= 0 -- Check if the ACK flag is set.

    -- If this is a SYN-ACK packet and the stream has an RST, set the custom field to true.
    if syn_flag and ack_flag and rst_stream_table[tcp_stream] then
        local subtree = tree:add(tcp_tagging_proto, tvb()) -- Add the protocol to the packet tree.
        subtree:add(f_tcp_syn_followed_by_rst, tvb(), true) -- Set the custom field value.
        pinfo.cols.info:append(" [SYN followed by RST]") -- Append a note to the packet info column.
    end
end


-- Dissector function for the custom protocol.
-- This function calls the two analysis functions to process each packet.
function tcp_tagging_proto.dissector(tvb, pinfo, tree)
    if ((tcp_stream_field() ~= nil) and (tcp_flags_field() ~= nil)) then
        analyze_packet_rst(pinfo, tvb, tree) -- Call the function to analyze RST packets.
        analyze_packet_tag(pinfo, tvb, tree) -- Call the function to tag SYN-ACK packets.
    end
end

-- Register the protocol as a postdisector.
-- This will ensure that the protocol is called after the standard dissectors.
register_postdissector(tcp_tagging_proto)