Big POST request to kubernetes cluster fails
Hey,
I wanted to get your opinion on this issue i'm having. So i've been recently setting up a k8s cluster, using metallb with bgp routing to set up routes from pfsense. Had a few issues with websockets because pfsense was a bit eager to close active connections, but set the pending packet timeout quite high and that seems to now be resolved.
However, I'm having issues with a long POST request not working. Small POST requests work fine, but when they are quite long, they start to fail. Unfortunately one of the apps I use does send a lot of data through a POST request, so it's broken on k8s. My k8s cluster is running over 3 VMs, and interestingly enough, if I CURL from the VM Host server, I don't get this issue. So I'm thinking it's either an issue with my hardware, or it's another issue with pfsense.
I'm using a HP ProCurve 1810-24G and a UniFi Switch 16 POE-150W, both of which have jumbo frames off, along with every NIC i've checked on the server and client. This is also happening from other clients and other VMs.
I've monitored it with wireshark, and this is what I see from a CURL request that represents a real life POST request from this app (but I'm posting it to HTTP-ECHO docker image to simplify things)
Posted a screenshot here (don't have enough points to upload directly) https://imgur.com/a/pUv0bMr
I'm no expert in TCP, but it seems as though the packets are getting lost - the client is 10.0.1.92 and the http echo app on the k8s cluster is 10.26.0.12.
I did another trace with a small POST (basically an empty body) which worked fine, so we can compare the two. (Under the sme imgurl link above)
I'm not sure what to try. My instinct is the issue is with pfsense, because the other issues I've had with this are also with pfsense, but I'm not sure what the issue is or how to fix it.
Any suggestions? I'm a Software guy and I'm starting to feel out of my depth here with debugging packets, and this issue is driving me crazy!
Thanks :)