issue with Mellanox NIC on recent Ubuntu

ConnectX

asked 2019-08-13 16:27:35 +0000

Yao Lin
1 ●3 ●1

This is not a wireshark issue.

Wireshark 2.x works with Mellanox ConnectX-4/ConnectX-5 in my Ubuntu 17.10 (kernel 4.13) PC.

However, it doesn't work in Ubuntu 18.04 (kernel 4.15) or Ubuntu 18.10 (kernel 4.18). It can capture the normal IP packets. But it won't show any RDMA (RoCEv2) packets. I also tried tcpdump and it shows the same issue. But once I replace Melloanox NICs with a QLogic NIC (41000 series), then wireshark is able to capture all the packets.

I am using inbox driver for all these NICs. There are technical reasons that prevents me from moving to Mellanox OFED driver.

I contacted Mellanox. But they didn't help and they disabled my account. :-(

So I am hoping I can get some help here.

edit retag flag offensive close merge delete

Comments

Unlikely to be a Wireshark issue, more likely to be the NIC drivers or libpcap which Wireshark uses to capture the traffic.

grahamb ( 2019-08-13 17:33:45 +0000 )edit

As mentioned in my 1st post, this is not a wireshark issue and I do think this is a Mellanox driver issue. I just hope some wireshark users have experience as to how to get around this issue w/o using Mellanox OFED driver.

My setup is

PC1 === Switch === PC2
           |
           |
           PC3

RoCE traffic is sent between PC1 and PC2. Wireshark is running in PC3. I configure the switch so that it makes a copy of all traffic between PC1/PC2 and forward them to PC3. So for PC3, these packets are not RDMA packets. They are just the normal IP packets, which shall be passed to the kernel by the NIC.

Yao Lin ( 2019-08-13 19:36:55 +0000 )edit

I configure the switch so that it makes a copy of all traffic between PC1/PC2 and forward them to PC3.

...which would mean that PC3 is plugged into the switch. Is that the case? I.e., are PC1, PC2, and PC3 all plugged into the switch, with the port into which PC3 is plugged being configured as a "mirror port" or whatever the switch vendor calls it?

Guy Harris ( 2019-08-13 20:54:51 +0000 )edit

Yes, all 3 PCs are connected to the switch. Somehow after I post the message, my message format was altered a bit and now it looks as if PC3 is not connected to the switch, which is certainly not true.

The feature in the switch I use is called "port monitoring" in the switch I use. This is pretty much the standard way to capture RDMA traffic.

Yao Lin ( 2019-08-13 21:06:15 +0000 )edit

I fixed your ASCII diagram by using the "code" format button.

grahamb ( 2019-08-13 22:14:20 +0000 )edit

see more comments

2 Answers

Sort by » oldest newest most voted

answered 2019-08-14 18:05:29 +0000

Yao Lin
1 ●3 ●1

Issue resolved.

Mellanox NICs in all my Ubuntu 18.x PCs were set to RoCEv2 mode, while NICs in my 17.10 PCs were running at default mode. I have verified that, if the NIC in the wireshark PC is running at default mode, we can always capture RDMA packets. If the NIC is running at RoCEv2 mode, then no RDMA packet will be captured. This has nothing to do with Ubuntu or kernel versions.

Thanks for the discussions. Some of your comments/questions prompt me to look into the NIC mode.

edit flag offensive delete link

add a comment

answered 2019-08-13 18:35:42 +0000

Guy Harris
19890 ●3 ●659 ●207

As Graham noted, and as

I also tried tcpdump and it shows the same issue.

indicates, this isn't a Wireshark issue. It's probably either an issue with the hardware (including on-board firmware), the driver, or, if there's special libpcap code for those adapters, the libpcap code. (A quick look at the Mellanox site, and the "E" in "ROCE", suggests that they're Ethernet adapters, not Infiniband adapters, so I suspect there might not be special libpcap code.)

Perhaps ROCE packets aren't passed, as raw Ethernet packets, to the host, and are processed in the adapter, which directly performs DMA operations to/from the host? If so, they won't show up if you try to capture on the adapter, unless there's a special mode in which copies of the raw Ethernet packets are passed to the host, and the driver enables that mode.

edit flag offensive delete link

add a comment