Why would a server not send a SYN/ACK packet in response to a SYN packet

Question

Lately, we've become aware of a TCP connection issue that is mostly limited to mac and Linux users who browse our websites.

From the user perspective, it presents itself as a really long connection time to our websites (>11 seconds).

We've managed to track down the technical signature of this problem, but can't figure out why it is happening or how to fix it.

Basically, what is happening is that the client's machine is sending the SYN packet to establish the TCP connection and the web server receives it, but does not respond with the SYN/ACK packet. After the client has sent many SYN packets, the server finally responds with a SYN/ACK packet and everything is fine for the remainder of the connection.

And, of course, the kicker to the problem: it is intermittent and does not happen all the time (though it does happen between 10-30% of the time)

We are using Fedora 12 Linux as the OS and Nginx as the web server.

Screenshot of wireshark analysis

Update:

Turning off window scaling on the client stopped the issue from happening. Now I just need a server side resolution (we can't make all the clients do this) :)

Final Update:

The solution was to turn off both TCP window scaling and TCP timestamps on our servers that are accessible to the public.

Do you have any acls or rules based on reverse DNS? You may need to look at more then just the connection between the client and the server. Perhaps a DNS lookup is timing out? — Zoredache, Feb 15 '11 at 23:28
@coredump: here's a screen shot of the wireshark analysis that shows the issue http://i.imgur.com/Bnzrm.png (couldn't figure out how to export just the stream....) — codemonkey, Feb 15 '11 at 23:49
@Zoredache: no, we don't have any acls or rules based on reverse DNS. This is a public facing webserver and we allow everyone to access it — codemonkey, Feb 15 '11 at 23:51
Just a hunch, but are you doing any kind of incoming connection rate-limiting on the server? Say, with iptables? — Steven Monday, Feb 15 '11 at 23:58
@Steven: no, we're not doing any connection rate-limiting. At least, not on purpose ;) — codemonkey, Feb 16 '11 at 00:08
What is the value of your net.ipv4.tcp_window_scaling sysctl? — coredump, Feb 16 '11 at 02:03
port 80 is the only port open to the public. this problem only happens when connecting from outside the LAN network — codemonkey, Feb 16 '11 at 15:20
Sometimes there's buggy software in middle: http://kerneltrap.org/node/6723 What is — poige, Feb 16 '11 at 17:53
If you actually solved this, you should come back and post your answer. (And I hope you got rid of Fedora 12 before now!) — Michael Hampton, Apr 05 '13 at 16:32

mcdizzle · Answer 1 · 2013-04-05T17:17:50.893

17

We had this exact same problem. Just disabling TCP timestamps solved the problem.

sysctl -w net.ipv4.tcp_timestamps=0

To make this change permanent, make an entry in /etc/sysctl.conf.

Be very careful about disabling the TCP Window Scale option. This option is important for providing maximum performance over the internet. Someone with a 10 megabit/sec connection will have a suboptimal transfer if the round trip time (basically same as ping) is more than 55 ms.

We really noticed this problem when there were multiple devices behind the same NAT. I suspect that the server might have been confused seeing timestamps from Android devices and OSX machines at the same time since they put completely different values in the timestamp fields.

edited Apr 05 '13 at 17:17

answered Apr 05 '13 at 16:26

mcdizzle

281

6

In case someone else ends up here through the same rabbit hole that I just went down: Before turning off TCP timestamps or window scaling, which may have severe performance consequences on a high-traffic link, check to see if tcp_tw_recycle is your problem: http://stackoverflow.com/questions/8893888/dropping-of-connections-with-tcp-tw-recycle – nephtes Oct 06 '14 at 20:43
First check that server and client have the correct time. – Nicolas VERHELST May 12 '22 at 05:21

score 16 · Answer 2 · edited Oct 03 '23 at 14:33

16

In my case the following command has fixed the problem with missing SYN/ACK replies from the Linux server:

sysctl -w net.ipv4.tcp_tw_recycle=0

I think this is more appropriate than disabling TCP timestamps, since TCP timestamps can be useful for high performance (PAWS, window scaling, etc).

The documentation on the tcp_tw_recycle explicitly states that it is not recommended to enable it, as many NAT routers preserve timestamps and this can cause PAWS (Protection Against Wrapping Sequence) to activate, as timestamps from the same source IP address can become inconsistent.

   tcp_tw_recycle (Boolean; default: disabled; since Linux 2.4)
          Enable fast recycling of TIME_WAIT sockets.  Enabling this
          option is not recommended for devices communicating with the
          general Internet or using NAT (Network Address Translation).
          Since some NAT gateways pass through IP timestamp values, one
          IP can appear to have non-increasing timestamps.  See RFC 1323
          (PAWS), RFC 6191.

edited Oct 03 '23 at 14:33

Community

1

answered Jun 27 '16 at 13:47

lav

361
3
8

1

good explanation at here: https://vincent.bernat.im/en/blog/2014-tcp-time-wait-state-linux
On the server side, do not enable net.ipv4.tcp_tw_recycle unless you are pretty sure you will never have NAT devices in the mix.
– Gnought Apr 06 '17 at 07:02
2

In my case, net.ipv4.tcp_tw_recycle is the real reason. Thanks. – bluearrow Dec 26 '17 at 03:10
2

tcp_tw_recycle has been removed in recent kernels. Is there another solution alike? @nephtes implies disabling timestamp hurts performance. – MappaM Mar 06 '19 at 14:28
1

Since tcp_tw_recycle has been removed, the problem should not occur again as it only happened with a non-default value of tcp_tw_recycle. – lav Mar 07 '19 at 09:01

score 5 · Answer 3 · edited Jun 11 '15 at 15:48

5

Just wondering, but why for the SYN packet (frame #539; the one that was accepted), the WS and TSV fields are missing in the "Info" column?

WS is TCP Window Scaling and TSV is Timestamp Value. Both of them are found under tcp.options field and Wireshark still should show them if they are present. Maybe Client TCP/IP stack resent different SYN packet on 8th attempt and that was the reason why it was suddenly acknowledged?

Could you provide us with frame 539 internal values? Does the SYN/ACK always comes for a SYN packet that does not have WS enabled?

edited Jun 11 '15 at 15:48

André Chalella

105

answered Feb 16 '11 at 00:29

user2138912

632

@Ansis: here's some screen shots for frame 539 details (had to do it in two parts): http://i.imgur.com/D84GC.png & http://i.imgur.com/4riq3.png – codemonkey Feb 16 '11 at 00:42
@codemonkey: Your 8th SYN packet seems to be different than the first seven SYN packets. Does the server respond with SYN/ACK to the client's SYN only when tcp.options field is of size 8 bytes (The first seven SYN packets probably have tcp.options of size 20 bytes.)? Can you disable TCP window scaling at the client side to see if the problem disappears? Seems like a problem with TCP/IP stack on the server side or misconfigured firewall somewhere... – user2138912 Feb 16 '11 at 00:52
@Ansis: yeah, I've been looking at that since you pointed it out and all the other SYN packets are 24 bytes. I will try disabling window scaling on the client and check back in with the results in the morning. – codemonkey Feb 16 '11 at 01:10
@Ansis: turning off windows scaling on the client stopped the issue from happening. Thanks! However, now I need to figure out how to fix this on the server side (since we can't make all our clients disable windows scaling) :) The server in question does have net.ipv4.tcp_windows_scaling = 1 – codemonkey Feb 16 '11 at 15:08
@Codemonkey: I agree that disabling WS on all clients is not a solution, but we at least have tracked the issue to WS/Packet Size issues. To further find the cause we should look into how your firewall is configured. Can you establish TCP connections with WS to different TCP ports? From different source IPs? – user2138912 Feb 16 '11 at 18:12
@Ansis: Yes, I can establish TCP connections with WS to the same TCP port from the problem client (the issue is intermittent). I can connect to other ports on the server from other hosts in the LAN network. With a VPN connection between the problem client and the server LAN, there are no connection issues. – codemonkey Feb 16 '11 at 18:37
@Ansis: Finally figured it out. I updated the post with the solution. Thanks for putting me on the right track! – codemonkey Feb 25 '11 at 15:22
@codemonkey: I see, but disabling WS is rather a workaround. I am wondering if there is buggy/misconfigured software on your Server that is causing this problem. Anyway I am glad that I helped you to get on the right track. – user2138912 Feb 25 '11 at 20:54

score 4 · Answer 4 · edited Sep 24 '12 at 23:45

4

We just ran into the exact same problem (really took quite a while to pin it to server not sending syn-ack).

"The solution was to turn off tcp windows scaling and tcp timestamps on our servers that are accessible to the public."

edited Sep 24 '12 at 23:45

sysadmin1138

134,165

answered Mar 18 '12 at 06:14

Alex Li

41

brablc · Answer 5 · 2018-10-15T19:35:51.470

3

The missing SYN/ACK could be caused by too low limits of your SYNFLOOD protection on firewall. It depends on how many connections to your server user creates. Using spdy would reduce the number of connections and could help in situation where turning net.ipv4.tcp_timestamps off does not help.

edited Oct 15 '18 at 19:35

answered May 20 '15 at 12:11

brablc

631

score 2 · Answer 6 · answered Feb 16 '11 at 01:15

2

To carry on what Ansis has stated, I've seen issues like this when the firewall doesn't support TCP Windows Scaling. What make/model firewall is between these two hosts?

answered Feb 16 '11 at 01:15

joeqwerty

110,860

The firewall is a Fedora 13 box using iptables. net.ipv4.tcp_windows_scaling is set to 1 on this machine also – codemonkey Feb 16 '11 at 15:12

score 1 · Answer 7 · answered Feb 16 '11 at 00:04

1

This is the behavior of a listening TCP socket when its backlog is full.

Ngnix allows the backlog argument to listen to be set in the configuration: http://wiki.nginx.org/HttpCoreModule#listen

listen 80 backlog=num

Try setting num to something larger than the default, like 1024.

I provide no guarantee that a full listen queue is actually your problem, but this is a good first thing to check.

answered Feb 16 '11 at 00:04

akramer

566

thanks for the tip. I will try it out. We've set the backlog at the OS level, but not explicitly in Nginx config. I'll update with the result. – codemonkey Feb 16 '11 at 00:11
it didn't change the behavior at all. Guess, it's not the problem? or the only problem... – codemonkey Feb 16 '11 at 00:25
1

application level backlog parameter controls size of queue for completed tcp connections i.e. 3-way handshake finished, i.e. syn-ack received - so it doesn't match OP situation – ygrek Oct 15 '15 at 21:59

score 1 · Answer 8 · answered Aug 28 '13 at 03:20

I just discovered that Linux TCP clients change their SYN packet after 3 tries, and remove the Window Scaling option. I guess the kernel developers figured that this is a common cause of connection failure in the Internet

It explains why these clients manage to connect after 11 seconds ( the window-less TCP SYN happens after 9 seconds in my brief test with default settings )

score 0 · Answer 9 · answered Oct 16 '18 at 20:36

0

I had a similar problem, but in my case it was the TCP checksum that was wrongly computed. The client was behind a veth and running ethtool -K veth0 rx off tx off did the trick.

answered Oct 16 '18 at 20:36

Baroudi Safwen

111
4

Why would a server not send a SYN/ACK packet in response to a SYN packet

9 Answers9

Linked

Related