Revision history [back]

TCP Retransmissions

This server's been running fine for over a year. Today we started initiating outgoing connections from it, and things haven't been great.

If I run:

time -p bash -c "
  for i in {1..100}; do 
    curl https://google.com -v
  done
"

Doesn't matter if it's google or something else. It'll stutter at least 4-5 times, usually while establishing TLS. It can take anywhere from 5 to timing out.

No.     Time           Source                Destination           Protocol Length Info
   1312 0.009835       32.23.109.22          74.125.200.113        TCP      68     49044 → 443 [SYN] Seq=0 Win=43650 Len=0 MSS=1455 SACK_PERM=1 WS=256
   1313 0.001671       74.125.200.113        32.23.109.22          TCP      68     443 → 49044 [SYN, ACK] Seq=0 Ack=1 Win=65535 Len=0 MSS=1430 SACK_PERM=1 WS=256
   1314 0.000026       32.23.109.22          74.125.200.113        TCP      56     49044 → 443 [ACK] Seq=1 Ack=1 Win=43776 Len=0
   1315 0.003958       32.23.109.22          74.125.200.113        TLSv1.3  573    Client Hello
   1316 0.001641       74.125.200.113        32.23.109.22          TCP      62     443 → 49044 [ACK] Seq=1 Ack=518 Win=66816 Len=0
   1317 0.000745       74.125.200.113        32.23.109.22          TLSv1.3  1486   Server Hello, Change Cipher Spec
   1318 0.000006       32.23.109.22          74.125.200.113        TCP      56     49044 → 443 [ACK] Seq=518 Ack=1431 Win=42496 Len=0
   1328 1.908929       74.125.200.113        32.23.109.22          TCP      1486   443 → 49044 [ACK] Seq=1431 Ack=518 Win=66816 Len=1430 [TCP segment of a reassembled PDU]
   1329 0.000039       32.23.109.22          74.125.200.113        TCP      56     49044 → 443 [ACK] Seq=518 Ack=2861 Win=42240 Len=0
   1348 2.787432       74.125.200.113        32.23.109.22          TCP      62     [TCP Previous segment not captured] 443 → 49044 [FIN, ACK] Seq=3843 Ack=518 Win=66816 Len=0
   1349 0.000029       32.23.109.22          74.125.200.113        TCP      68     [TCP Dup ACK 1329#1] 49044 → 443 [ACK] Seq=518 Ack=2861 Win=42240 Len=0 SLE=3843 SRE=3844
   1350 0.001707       74.125.200.113        32.23.109.22          TCP      1038   [TCP Retransmission] 443 → 49044 [PSH, ACK] Seq=2861 Ack=518 Win=66816 Len=982
   1351 0.000037       32.23.109.22          74.125.200.113        TCP      56     49044 → 443 [ACK] Seq=518 Ack=3844 Win=42240 Len=0
   1352 0.001874       32.23.109.22          74.125.200.113        TLSv1.3  136    Change Cipher Spec, Application Data
   1353 0.000410       32.23.109.22          74.125.200.113        TLSv1.3  102    Application Data
   1354 0.000018       32.23.109.22          74.125.200.113        TLSv1.3  105    Application Data
   1355 0.000099       32.23.109.22          74.125.200.113        TLSv1.3  149    Application Data, Application Data
   1356 0.000531       32.23.109.22          74.125.200.113        TLSv1.3  80     Application Data
   1357 0.000621       74.125.200.113        32.23.109.22          TCP      62     443 → 49044 [RST] Seq=3844 Win=0 Len=0
   1358 0.000400       74.125.200.113        32.23.109.22          TCP      62     443 → 49044 [RST] Seq=3844 Win=0 Len=0
   1359 0.000036       74.125.200.113        32.23.109.22          TCP      62     443 → 49044 [RST] Seq=3844 Win=0 Len=0
   1360 0.000605       74.125.200.113        32.23.109.22          TCP      62     443 → 49044 [RST] Seq=3844 Win=0 Len=0

Wondering if anyone can get any insight from this as I've pretty much hit a wall.

TCP Retransmissions

Update: Giving a non-https capture. This one was really bad. It was a single CURL to http://www.bom.gov.au/

No.     Time           Source                Destination           Protocol Length Info
      1 0.000000       32.23.109.22          23.54.57.70           TCP      66     24434 → 80 [SYN] Seq=0 Win=42340 Len=0 MSS=1460 SACK_PERM=1 WS=256
      2 0.000998       23.54.57.70           32.23.109.22          TCP      66     80 → 24434 [SYN, ACK] Seq=0 Ack=1 Win=29200 Len=0 MSS=1460 SACK_PERM=1 WS=128
      3 0.000051       32.23.109.22          23.54.57.70           TCP      54     24434 → 80 [ACK] Seq=1 Ack=1 Win=42496 Len=0
      4 0.000119       32.23.109.22          23.54.57.70           HTTP     132    GET / HTTP/1.1 
      5 0.000990       23.54.57.70           32.23.109.22          TCP      60     80 → 24434 [ACK] Seq=1 Ack=79 Win=29312 Len=0
      6 0.107525       23.54.57.70           32.23.109.22          HTTP     2974   [TCP Previous segment not captured] Continuation
      7 0.000030       32.23.109.22          23.54.57.70           TCP      66     [TCP Dup ACK 3#1] 24434 → 80 [ACK] Seq=79 Ack=1 Win=42496 Len=0 SLE=7301 SRE=10221
      8 0.004211       23.54.57.70           32.23.109.22          TCP      1514   [TCP Retransmission] 80 → 24434 [ACK] Seq=1 Ack=79 Win=29312 Len=1460
      9 0.000038       32.23.109.22          23.54.57.70           TCP      66     24434 → 80 [ACK] Seq=79 Ack=1461 Win=41216 Len=0 SLE=7301 SRE=10221
     10 12.953972      23.54.57.70           32.23.109.22          TCP      1262   [TCP Retransmission] 80 → 24434 [ACK] Seq=1461 Ack=79 Win=29312 Len=1208
     11 0.000041       32.23.109.22          23.54.57.70           TCP      66     24434 → 80 [ACK] Seq=79 Ack=2669 Win=40704 Len=0 SLE=7301 SRE=10221
     12 0.001026       23.54.57.70           32.23.109.22          HTTP     1262   [TCP Previous segment not captured] Continuation
     13 0.000000       23.54.57.70           32.23.109.22          HTTP     2470   [TCP Previous segment not captured] Continuation
     14 0.000032       32.23.109.22          23.54.57.70           TCP      74     [TCP Dup ACK 11#1] 24434 → 80 [ACK] Seq=79 Ack=2669 Win=40704 Len=0 SLE=30661 SRE=31869 SLE=7301 SRE=10221
     15 0.000011       32.23.109.22          23.54.57.70           TCP      82     [TCP Dup ACK 11#2] 24434 → 80 [ACK] Seq=79 Ack=2669 Win=40704 Len=0 SLE=33077 SRE=35493 SLE=30661 SRE=31869 SLE=7301 SRE=10221
     16 0.001006       23.54.57.70           32.23.109.22          TCP      306    [TCP Fast Retransmission] 80 → 24434 [ACK] Seq=2669 Ack=79 Win=29312 Len=252 [TCP segment of a reassembled PDU]
     17 0.000038       32.23.109.22          23.54.57.70           TCP      82     24434 → 80 [ACK] Seq=79 Ack=2921 Win=40704 Len=0 SLE=33077 SRE=35493 SLE=30661 SRE=31869 SLE=7301 SRE=10221
     18 0.205820       23.54.57.70           32.23.109.22          TCP      1262   [TCP Retransmission] 80 → 24434 [ACK] Seq=2921 Ack=79 Win=29312 Len=1208
     19 0.000039       32.23.109.22          23.54.57.70           TCP      82     24434 → 80 [ACK] Seq=79 Ack=4129 Win=39680 Len=0 SLE=33077 SRE=35493 SLE=30661 SRE=31869 SLE=7301 SRE=10221
     20 0.001001       23.54.57.70           32.23.109.22          TCP      1262   [TCP Retransmission] 80 → 24434 [ACK] Seq=4129 Ack=79 Win=29312 Len=1208
     21 0.000000       23.54.57.70           32.23.109.22          TCP      306    [TCP Retransmission] 80 → 24434 [ACK] Seq=11429 Ack=79 Win=29312 Len=252
     22 0.000001       23.54.57.70           32.23.109.22          TCP      1262   [TCP Retransmission] 80 → 24434 [ACK] Seq=11681 Ack=79 Win=29312 Len=1208
     23 0.000000       23.54.57.70           32.23.109.22          TCP      1262   [TCP Retransmission] 80 → 24434 [ACK] Seq=18929 Ack=79 Win=29312 Len=1208
     24 0.000000       23.54.57.70           32.23.109.22          TCP      862    [TCP Retransmission] 80 → 24434 [ACK] Seq=22553 Ack=79 Win=29312 Len=808
     25 0.000044       32.23.109.22          23.54.57.70           TCP      82     24434 → 80 [ACK] Seq=79 Ack=5337 Win=38656 Len=0 SLE=33077 SRE=35493 SLE=30661 SRE=31869 SLE=7301 SRE=10221
     26 0.000011       32.23.109.22          23.54.57.70           TCP      90     [TCP Dup ACK 25#1] 24434 → 80 [ACK] Seq=79 Ack=5337 Win=38656 Len=0 SLE=11429 SRE=11681 SLE=33077 SRE=35493 SLE=30661 SRE=31869 SLE=7301 SRE=10221
     27 0.000007       32.23.109.22          23.54.57.70           TCP      90     [TCP Dup ACK 25#2] 24434 → 80 [ACK] Seq=79 Ack=5337 Win=38656 Len=0 SLE=11429 SRE=12889 SLE=33077 SRE=35493 SLE=30661 SRE=31869 SLE=7301 SRE=10221
     28 0.000006       32.23.109.22          23.54.57.70           TCP      90     [TCP Dup ACK 25#3] 24434 → 80 [ACK] Seq=79 Ack=5337 Win=38656 Len=0 SLE=18929 SRE=20137 SLE=11429 SRE=12889 SLE=33077 SRE=35493 SLE=30661 SRE=31869
     29 0.000006       32.23.109.22          23.54.57.70           TCP      90     [TCP Dup ACK 25#4] 24434 → 80 [ACK] Seq=79 Ack=5337 Win=38656 Len=0 SLE=22553 SRE=23361 SLE=18929 SRE=20137 SLE=11429 SRE=12889 SLE=33077 SRE=35493
     30 0.000131       23.54.57.70           32.23.109.22          TCP      1262   [TCP Retransmission] 80 → 24434 [ACK] Seq=23361 Ack=79 Win=29312 Len=1208
     31 0.000030       32.23.109.22          23.54.57.70           TCP      90     [TCP Dup ACK 25#5] 24434 → 80 [ACK] Seq=79 Ack=5337 Win=38656 Len=0 SLE=22553 SRE=24569 SLE=18929 SRE=20137 SLE=11429 SRE=12889 SLE=33077 SRE=35493
     32 0.000839       23.54.57.70           32.23.109.22          TCP      1062   [TCP Retransmission] 80 → 24434 [ACK] Seq=28193 Ack=79 Win=29312 Len=1008
     33 0.000000       23.54.57.70           32.23.109.22          TCP      1262   [TCP Retransmission] 80 → 24434 [ACK] Seq=29201 Ack=79 Win=29312 Len=1208
     34 0.000000       23.54.57.70           32.23.109.22          TCP      1262   [TCP Retransmission] 80 → 24434 [ACK] Seq=31869 Ack=79 Win=29312 Len=1208
     35 0.000034       32.23.109.22          23.54.57.70           TCP      90     [TCP Dup ACK 25#6] 24434 → 80 [ACK] Seq=79 Ack=5337 Win=38656 Len=0 SLE=28193 SRE=29201 SLE=22553 SRE=24569 SLE=18929 SRE=20137 SLE=11429 SRE=12889
     36 0.000011       32.23.109.22          23.54.57.70           TCP      90     [TCP Dup ACK 25#7] 24434 → 80 [ACK] Seq=79 Ack=5337 Win=38656 Len=0 SLE=28193 SRE=30409 SLE=22553 SRE=24569 SLE=18929 SRE=20137 SLE=11429 SRE=12889
     37 0.000006       32.23.109.22          23.54.57.70           TCP      90     [TCP Dup ACK 25#8] 24434 → 80 [ACK] Seq=79 Ack=5337 Win=38656 Len=0 SLE=31869 SRE=33077 SLE=28193 SRE=30409 SLE=22553 SRE=24569 SLE=18929 SRE=20137
     38 0.000006       23.54.57.70           32.23.109.22          TCP      1262   [TCP Retransmission] 80 → 24434 [ACK] Seq=10221 Ack=79 Win=29312 Len=1208
     39 0.000009       32.23.109.22          23.54.57.70           TCP      90     [TCP Dup ACK 25#9] 24434 → 80 [ACK] Seq=79 Ack=5337 Win=38656 Len=0 SLE=10221 SRE=11429 SLE=31869 SRE=33077 SLE=28193 SRE=30409 SLE=22553 SRE=24569
     40 0.000116       23.54.57.70           32.23.109.22          TCP      1262   [TCP Retransmission] 80 → 24434 [ACK] Seq=12889 Ack=79 Win=29312 Len=1208
     41 0.000030       32.23.109.22          23.54.57.70           TCP      90     [TCP Dup ACK 25#10] 24434 → 80 [ACK] Seq=79 Ack=5337 Win=38656 Len=0 SLE=12889 SRE=14097 SLE=10221 SRE=11429 SLE=31869 SRE=33077 SLE=28193 SRE=30409
     42 0.000841       23.54.57.70           32.23.109.22          TCP      2470   [TCP Retransmission] 80 → 24434 [ACK] Seq=15305 Ack=79 Win=29312 Len=2416
     43 0.000030       32.23.109.22          23.54.57.70           TCP      90     [TCP Dup ACK 25#11] 24434 → 80 [ACK] Seq=79 Ack=5337 Win=38656 Len=0 SLE=15305 SRE=17721 SLE=12889 SRE=14097 SLE=10221 SRE=11429 SLE=31869 SRE=33077
     44 0.000114       23.54.57.70           32.23.109.22          TCP      1262   [TCP Retransmission] 80 → 24434 [ACK] Seq=17721 Ack=79 Win=29312 Len=1208
     45 0.000030       32.23.109.22          23.54.57.70           TCP      90     [TCP Dup ACK 25#12] 24434 → 80 [ACK] Seq=79 Ack=5337 Win=38656 Len=0 SLE=15305 SRE=18929 SLE=12889 SRE=14097 SLE=10221 SRE=11429 SLE=31869 SRE=33077
     46 0.000989       23.54.57.70           32.23.109.22          TCP      306    [TCP Retransmission] 80 → 24434 [ACK] Seq=30409 Ack=79 Win=29312 Len=252
     47 0.000029       32.23.109.22          23.54.57.70           TCP      90     [TCP Dup ACK 25#13] 24434 → 80 [ACK] Seq=79 Ack=5337 Win=38656 Len=0 SLE=30409 SRE=30661 SLE=15305 SRE=18929 SLE=12889 SRE=14097 SLE=10221 SRE=11429
     48 0.000979       23.54.57.70           32.23.109.22          HTTP     1195   [TCP Previous segment not captured] Continuation
     49 0.000029       32.23.109.22          23.54.57.70           TCP      90     [TCP Dup ACK 25#14] 24434 → 80 [ACK] Seq=79 Ack=5337 Win=38656 Len=0 SLE=37909 SRE=39050 SLE=30409 SRE=30661 SLE=15305 SRE=18929 SLE=12889 SRE=14097
     50 0.001004       23.54.57.70           32.23.109.22          HTTP     2018   [TCP Fast Retransmission] Continuation
     51 0.000040       32.23.109.22          23.54.57.70           TCP      82     24434 → 80 [ACK] Seq=79 Ack=14097 Win=29952 Len=0 SLE=37909 SRE=39050 SLE=30409 SRE=30661 SLE=15305 SRE=18929
     52 0.000954       23.54.57.70           32.23.109.22          TCP      1262   [TCP Retransmission] 80 → 24434 [ACK] Seq=25777 Ack=79 Win=29312 Len=1208
     53 0.000026       32.23.109.22          23.54.57.70           TCP      90     [TCP Dup ACK 51#1] 24434 → 80 [ACK] Seq=79 Ack=14097 Win=29952 Len=0 SLE=25777 SRE=26985 SLE=37909 SRE=39050 SLE=30409 SRE=30661 SLE=15305 SRE=18929
     54 0.001042       23.54.57.70           32.23.109.22          TCP      1262   [TCP Retransmission] 80 → 24434 [ACK] Seq=14097 Ack=79 Win=29312 Len=1208
     55 0.000040       32.23.109.22          23.54.57.70           TCP      82     24434 → 80 [ACK] Seq=79 Ack=20137 Win=27136 Len=0 SLE=25777 SRE=26985 SLE=37909 SRE=39050 SLE=30409 SRE=30661
     56 0.000143       23.54.57.70           32.23.109.22          TCP      1262   [TCP Retransmission] 80 → 24434 [ACK] Seq=21345 Ack=79 Win=29312 Len=1208
     57 0.000030       32.23.109.22          23.54.57.70           TCP      90     [TCP Window Update] 24434 → 80 [ACK] Seq=79 Ack=20137 Win=30976 Len=0 SLE=21345 SRE=22553 SLE=25777 SRE=26985 SLE=37909 SRE=39050 SLE=30409 SRE=30661
     58 0.000999       23.54.57.70           32.23.109.22          TCP      1262   [TCP Retransmission] 80 → 24434 [ACK] Seq=20137 Ack=79 Win=29312 Len=1208
     59 0.000040       32.23.109.22          23.54.57.70           TCP      82     24434 → 80 [ACK] Seq=79 Ack=24569 Win=30208 Len=0 SLE=25777 SRE=26985 SLE=37909 SRE=39050 SLE=30409 SRE=30661
     60 3.110323       23.54.57.70           32.23.109.22          TCP      1262   [TCP Retransmission] 80 → 24434 [ACK] Seq=24569 Ack=79 Win=29312 Len=1208
     61 0.000037       32.23.109.22          23.54.57.70           TCP      74     24434 → 80 [ACK] Seq=79 Ack=26985 Win=33280 Len=0 SLE=37909 SRE=39050 SLE=30409 SRE=30661
     62 0.001017       23.54.57.70           32.23.109.22          TCP      1262   [TCP Retransmission] 80 → 24434 [ACK] Seq=35493 Ack=79 Win=29312 Len=1208
     63 0.000027       32.23.109.22          23.54.57.70           TCP      82     [TCP Window Update] 24434 → 80 [ACK] Seq=79 Ack=26985 Win=34304 Len=0 SLE=35493 SRE=36701 SLE=37909 SRE=39050 SLE=30409 SRE=30661
     64 23.294869      23.54.57.70           32.23.109.22          TCP      1262   [TCP Retransmission] 80 → 24434 [ACK] Seq=26985 Ack=79 Win=29312 Len=1208
     65 0.000015       32.23.109.22          23.54.57.70           TCP      66     24434 → 80 [ACK] Seq=79 Ack=36701 Win=33536 Len=0 SLE=37909 SRE=39050
     77 31.594320      32.23.109.22          23.54.57.70           TCP      66     [TCP Keep-Alive] 24434 → 80 [ACK] Seq=78 Ack=36701 Win=42240 Len=0 SLE=37909 SRE=39050
     78 0.001010       23.54.57.70           32.23.109.22          TCP      60     [TCP Keep-Alive ACK] 80 → 24434 [ACK] Seq=39050 Ack=79 Win=29312 Len=0
     79 61.438983      32.23.109.22          23.54.57.70           TCP      66     [TCP Keep-Alive] 24434 → 80 [ACK] Seq=78 Ack=36701 Win=42240 Len=0 SLE=37909 SRE=39050
     80 0.000983       23.54.57.70           32.23.109.22          TCP      60     80 → 24434 [RST] Seq=36701 Win=0 Len=0

ORIGINAL POST:

This server's been running fine for over a year. Today we started initiating outgoing connections from it, and things haven't been great.

If I run:

time -p bash -c "
  for i in {1..100}; do 
    curl https://google.com -v
  done
"

Doesn't matter if it's google or something else. It'll stutter at least 4-5 times, usually while establishing TLS. It can take anywhere from 5 to timing out.

No.     Time           Source                Destination           Protocol Length Info
   1312 0.009835       32.23.109.22          74.125.200.113        TCP      68     49044 → 443 [SYN] Seq=0 Win=43650 Len=0 MSS=1455 SACK_PERM=1 WS=256
   1313 0.001671       74.125.200.113        32.23.109.22          TCP      68     443 → 49044 [SYN, ACK] Seq=0 Ack=1 Win=65535 Len=0 MSS=1430 SACK_PERM=1 WS=256
   1314 0.000026       32.23.109.22          74.125.200.113        TCP      56     49044 → 443 [ACK] Seq=1 Ack=1 Win=43776 Len=0
   1315 0.003958       32.23.109.22          74.125.200.113        TLSv1.3  573    Client Hello
   1316 0.001641       74.125.200.113        32.23.109.22          TCP      62     443 → 49044 [ACK] Seq=1 Ack=518 Win=66816 Len=0
   1317 0.000745       74.125.200.113        32.23.109.22          TLSv1.3  1486   Server Hello, Change Cipher Spec
   1318 0.000006       32.23.109.22          74.125.200.113        TCP      56     49044 → 443 [ACK] Seq=518 Ack=1431 Win=42496 Len=0
   1328 1.908929       74.125.200.113        32.23.109.22          TCP      1486   443 → 49044 [ACK] Seq=1431 Ack=518 Win=66816 Len=1430 [TCP segment of a reassembled PDU]
   1329 0.000039       32.23.109.22          74.125.200.113        TCP      56     49044 → 443 [ACK] Seq=518 Ack=2861 Win=42240 Len=0
   1348 2.787432       74.125.200.113        32.23.109.22          TCP      62     [TCP Previous segment not captured] 443 → 49044 [FIN, ACK] Seq=3843 Ack=518 Win=66816 Len=0
   1349 0.000029       32.23.109.22          74.125.200.113        TCP      68     [TCP Dup ACK 1329#1] 49044 → 443 [ACK] Seq=518 Ack=2861 Win=42240 Len=0 SLE=3843 SRE=3844
   1350 0.001707       74.125.200.113        32.23.109.22          TCP      1038   [TCP Retransmission] 443 → 49044 [PSH, ACK] Seq=2861 Ack=518 Win=66816 Len=982
   1351 0.000037       32.23.109.22          74.125.200.113        TCP      56     49044 → 443 [ACK] Seq=518 Ack=3844 Win=42240 Len=0
   1352 0.001874       32.23.109.22          74.125.200.113        TLSv1.3  136    Change Cipher Spec, Application Data
   1353 0.000410       32.23.109.22          74.125.200.113        TLSv1.3  102    Application Data
   1354 0.000018       32.23.109.22          74.125.200.113        TLSv1.3  105    Application Data
   1355 0.000099       32.23.109.22          74.125.200.113        TLSv1.3  149    Application Data, Application Data
   1356 0.000531       32.23.109.22          74.125.200.113        TLSv1.3  80     Application Data
   1357 0.000621       74.125.200.113        32.23.109.22          TCP      62     443 → 49044 [RST] Seq=3844 Win=0 Len=0
   1358 0.000400       74.125.200.113        32.23.109.22          TCP      62     443 → 49044 [RST] Seq=3844 Win=0 Len=0
   1359 0.000036       74.125.200.113        32.23.109.22          TCP      62     443 → 49044 [RST] Seq=3844 Win=0 Len=0
   1360 0.000605       74.125.200.113        32.23.109.22          TCP      62     443 → 49044 [RST] Seq=3844 Win=0 Len=0

Wondering if anyone can get any insight from this as I've pretty much hit a wall.