This is a static archive of our old Q&A Site. Please post any new questions and answers at ask.wireshark.org.

A Question regarding weird Duplicate ICMP response

0

Hi Guys,

I tried to use winpcap to write a software running on PC (with 2 ethernet cards) to emulate a NAT. In that way, I could use a pesudo external IP address to access an internal IP. The architecture is as attached graph. However when I tried to ping the pesudo external IP address, I always get a lot of DUPLICATE ICMP response, does anyone knows why is that?

I do a ping from 146.208.243.51 -> 146.208.233.8, on the NAT server, the IP address and MAC address is converted to internal LAN's IP address/MAC address; and when the PC2 reply the ping, the NAT server will convert those IP/MAC address back to external IP/MAC address. However, when the ping response packet send back to the external network, I got following output. I've also tried if the PC3 stay in the same subnet as the NAT server, it seems everything work ok. But if the IP packet goes to Router/Gateway, this strange behavior is observed.alt text

64 bytes from 146.208.233.8: icmp_seq=1 ttl=62 time=4.09 ms
64 bytes from 146.208.233.8: icmp_seq=1 ttl=61 time=4.39 ms (DUP!)
64 bytes from 146.208.233.8: icmp_seq=1 ttl=60 time=4.60 ms (DUP!)
64 bytes from 146.208.233.8: icmp_seq=1 ttl=59 time=4.69 ms (DUP!)
64 bytes from 146.208.233.8: icmp_seq=1 ttl=58 time=4.73 ms (DUP!)
64 bytes from 146.208.233.8: icmp_seq=1 ttl=57 time=4.82 ms (DUP!)
64 bytes from 146.208.233.8: icmp_seq=1 ttl=56 time=4.85 ms (DUP!)
64 bytes from 146.208.233.8: icmp_seq=1 ttl=55 time=4.94 ms (DUP!)
64 bytes from 146.208.233.8: icmp_seq=1 ttl=54 time=5.08 ms (DUP!)
64 bytes from 146.208.233.8: icmp_seq=1 ttl=53 time=5.27 ms (DUP!)
64 bytes from 146.208.233.8: icmp_seq=1 ttl=52 time=5.45 ms (DUP!)
64 bytes from 146.208.233.8: icmp_seq=1 ttl=51 time=5.65 ms (DUP!)
64 bytes from 146.208.233.8: icmp_seq=1 ttl=50 time=5.82 ms (DUP!)
64 bytes from 146.208.233.8: icmp_seq=1 ttl=49 time=5.95 ms (DUP!)
64 bytes from 146.208.233.8: icmp_seq=1 ttl=48 time=6.03 ms (DUP!)
64 bytes from 146.208.233.8: icmp_seq=1 ttl=47 time=6.12 ms (DUP!)
64 bytes from 146.208.233.8: icmp_seq=1 ttl=46 time=6.20 ms (DUP!)
64 bytes from 146.208.233.8: icmp_seq=1 ttl=45 time=6.28 ms (DUP!)
64 bytes from 146.208.233.8: icmp_seq=1 ttl=44 time=6.38 ms (DUP!)
64 bytes from 146.208.233.8: icmp_seq=1 ttl=43 time=6.45 ms (DUP!)
64 bytes from 146.208.233.8: icmp_seq=1 ttl=42 time=6.52 ms (DUP!)
64 bytes from 146.208.233.8: icmp_seq=1 ttl=41 time=6.56 ms (DUP!)
64 bytes from 146.208.233.8: icmp_seq=1 ttl=40 time=6.65 ms (DUP!)
64 bytes from 146.208.233.8: icmp_seq=1 ttl=39 time=6.69 ms (DUP!)
64 bytes from 146.208.233.8: icmp_seq=1 ttl=38 time=6.78 ms (DUP!)
64 bytes from 146.208.233.8: icmp_seq=1 ttl=37 time=6.81 ms (DUP!)
64 bytes from 146.208.233.8: icmp_seq=1 ttl=36 time=6.90 ms (DUP!)
64 bytes from 146.208.233.8: icmp_seq=1 ttl=35 time=6.94 ms (DUP!)
64 bytes from 146.208.233.8: icmp_seq=1 ttl=34 time=7.02 ms (DUP!)
64 bytes from 146.208.233.8: icmp_seq=1 ttl=33 time=7.06 ms (DUP!)
64 bytes from 146.208.233.8: icmp_seq=1 ttl=32 time=7.15 ms (DUP!)
64 bytes from 146.208.233.8: icmp_seq=1 ttl=31 time=7.19 ms (DUP!)
64 bytes from 146.208.233.8: icmp_seq=1 ttl=30 time=7.27 ms (DUP!)
64 bytes from 146.208.233.8: icmp_seq=1 ttl=29 time=7.31 ms (DUP!)
64 bytes from 146.208.233.8: icmp_seq=1 ttl=28 time=7.40 ms (DUP!)
64 bytes from 146.208.233.8: icmp_seq=1 ttl=27 time=7.44 ms (DUP!)
64 bytes from 146.208.233.8: icmp_seq=1 ttl=26 time=7.52 ms (DUP!)
64 bytes from 146.208.233.8: icmp_seq=1 ttl=25 time=7.56 ms (DUP!)
64 bytes from 146.208.233.8: icmp_seq=1 ttl=24 time=7.65 ms (DUP!)
64 bytes from 146.208.233.8: icmp_seq=1 ttl=23 time=7.68 ms (DUP!)
64 bytes from 146.208.233.8: icmp_seq=1 ttl=22 time=7.77 ms (DUP!)
64 bytes from 146.208.233.8: icmp_seq=1 ttl=21 time=7.80 ms (DUP!)
64 bytes from 146.208.233.8: icmp_seq=1 ttl=20 time=7.89 ms (DUP!)

Does anybody knows why?

Thanks in advance

asked 02 Dec '13, 00:53

Jianhui's gravatar image

Jianhui
6224
accept rate: 0%

edited 02 Dec '13, 01:28

Kurt%20Knochner's gravatar image

Kurt Knochner ♦
24.8k1039237


One Answer:

1

as you can see, the TTL is 'counting' down. That's usually a sign for a routing loop. However, as you wrote the NAT code yourself, it could also be a bug in your code.

It's hard to diagnose without a capture file, taken at the right place(s). Can you post one at Google drive, dropbox, cloudshark.org, or mega.co.nz?

You should take a capture between PC2 -> PC(NAT) and between PC(NAT) -> GW1 at the same time.

Regards
Kurt

answered 02 Dec '13, 01:39

Kurt%20Knochner's gravatar image

Kurt Knochner ♦
24.8k1039237
accept rate: 15%

edited 02 Dec '13, 03:30

Hi Kurt,

Thanks for your quick response. I capture a log from the NAT server to capture wireshark log on both ethernet interface.The log is put on https://dl.dropboxusercontent.com/u/80123665/icmp.pcapng . You could use icmp filter to see ping request/response.

For example, You could see:

msg No. 257, ping request sent from 146.208.243.51 -> 146.208.233.8

msg No. 299, the ping request has been converted into internal LAN, IP/MAC address has been converted to 10.168.0.100 -> 10.168.0.2

msg No. 302, the ping response sent back from 10.168.0.2 -> 10.168.0.100

msg No. 262, the ping response has been converted back into external LAN, IP/MAC address has been converted to 146.208.233.8 -> 146.208.243.51

After that, I got 2 redirect message (msg 275,277) from 146.208.234.171 and 146.208.235.84, does that mean the router are suggesting other gateway? And I got a ICMP time-tolive exceeded packet in message No.313.

In this case, for internal LAN, I use the internal ethernet LAN MAC ID for both 10.168.0.1 and 10.168.0.100; for external LAN, I use the external LAN MAC ID for both 146.208.235.150 and 146.208.233.8 IP address. Is this an issue for the router?

Thanks a lot, Jianhui

(02 Dec '13, 20:15) Jianhui

A clarification of the system diagram is the PC3 is actually a Linux server instead of a PC running Microsoft Windows 7.

(02 Dec '13, 20:22) Jianhui
1

In Frame #257 the ICMP request is coming from a Cisco MAC address (I guess GW1). Then in Frame #262, the ICMP response is going to the MAC of a Dell system (Dell_d9:a5:d0 (00:1c:23:d9:a5:d0)). Then you get two ICMP redirects (Frame #275,277) from systems that are not included in your picture (IP: 146.208.234.171 and 146.208.235.84) and both have a different MAC address than the Dell address !??!

Spoiler: I guess, there is something terribly wrong with your network setup and most certainly there is a routing problem, as I suspect a routing loop somewhere ;-)

Please post the output of the following commands on both systems: PC(NAT) and GW1

netstat -nr
ipconfig -a
tracert -d 146.208.243.51

and the equivalent output on your Cisco device (GW1?).

Please identify the systems that sent the ICMP redirects ( 146.208.234.171 and 146.208.235.84) and check how they are involved.

Next thing: The ICMP request/reply is first shown completely on the external side of PC(NAT): Frames: #257,#262 and only after that on the internal side: #299,#302. That's not how NAT works.

On a a NAT device, I would have expected the following packet sequence:

  • External: ICMP request
  • Internal: ICMP request (address translated to internal)
  • Internal: ICMP reply
  • External: ICMP reply (address translated to external)

Did you configure the NAT IP address (146.208.233.8) on PC(NAT) as a secondary address?

To me it looks like you did that and the IP stack of the 'NAT' system answers the external ICMP requests (see ICMP packet sequence in the capture file). Then your WinPcap 'NAT code' takes the request and sends it to the internal side, which is then answered by the internal PC.

If that is true: That's not a good idea.

My suggestion:

  1. Fix your network setup: Check routing, etc.
  2. Think about how NAT should work
  3. Fix your IP configuration on PC(NAT)
  4. Maybe also required: Fix your WinPcap NAT code
  5. Repeat tests
(03 Dec '13, 02:42) Kurt Knochner ♦

Hi Kurt,

Thanks for your detailed response. I think I understand the issue right now.

I overlook the MAC address of the incoming ping request from external LAN, that MAC address (GW1's MAC address) is the MAC address that I should used for the ping response. However when I write NAT code, I convert that ping response Packet's Dest MAC address to the final destination's MAC address (PC3's MAC address instead of GW1's MAC address). It means I sent back a packet with a MAC address that cannot reached in the sub-net. That cause problem to the router while result in the Duplicate ping response.(But I'm not sure how the loopback route is created by the router, maybe you could shed some light on that ;-) )

As for the NAT problem you point out, I think that is caused by wireshark' logging algorithm. It puzzled me too when I see the log the first time. Then I found if you reorder the logging packets by time (instead of by sequence number) you would see the expected packet sequence:

External: ICMP request

Internal: ICMP request (address translated to internal)

Internal: ICMP reply

External: ICMP reply (address translated to external)

Thanks again for your great help on this, It is really very helpful! Jianhui

(03 Dec '13, 17:47) Jianhui

It means I sent back a packet with a MAC address that cannot reached in the sub-net.

Yes, and therefore no other system should 'feel' itself responsible for that packet. Why you get the ICMP redirects is unclear to me and I cannot give you any good explanation, as I don't know the rest of your infrastructure. I leave it up to you to figure out what's going on ;-)

Then I found if you reorder the logging packets by time (instead of by sequence number) you would see the expected packet sequence:

you are right. I did not check that. That's probably related to the way WinPcap (or dumpcap) buffers frames internally before they get flushed out to a capture file, especially if you capture on two interfaces at the same time

Note to myself: Don't assume anything ;-))

Thanks again for your great help on this, It is really very helpful! Jianhui

You're welcome, especially as these kinds of problem are rather interesting to diagnose :-)

Hint: If a supplied answer resolves your question can you please "accept" it by clicking the checkmark icon next to it. This highlights good answers for the benefit of subsequent users with the same or similar questions. For extra points you can up vote the answer (thumb up).

(04 Dec '13, 02:39) Kurt Knochner ♦