Additional Kernel Tweaks for IPv4 Broadband

High-speed broadband connections on Linux often underperform out of the box because the default TCP/IP kernel parameters are tuned for modest links. Adjusting a handful of values under /proc/sys/net/ can unlock significantly better throughput on high-latency, high-bandwidth connections without any hardware changes.

Locating the TCP/IP Parameters

All TCP/IP tuning parameters live under /proc/sys/net/... (assuming a 2.1+ kernel). This location is volatile, and changes are reset at reboot. Here is a list of the most important tuning parameters and what they control:

• /proc/sys/net/core/rmem_max - Maximum TCP Receive Window
• /proc/sys/net/core/wmem_max - Maximum TCP Send Window
• /proc/sys/net/ipv4/tcp_rmem - Memory reserved for TCP receive buffers
• /proc/sys/net/ipv4/tcp_wmem - Memory reserved for TCP send buffers
• /proc/sys/net/ipv4/tcp_timestamps - Timestamps (RFC 1323) add 12 bytes to the TCP header
• /proc/sys/net/ipv4/tcp_sack - TCP Selective Acknowledgements. They can reduce retransmissions, however they make servers more prone to DDoS attacks and increase CPU utilization
• /proc/sys/net/ipv4/tcp_window_scaling - Support for large TCP Windows (RFC 1323). Needs to be set to 1 if the Max TCP Window is over 65535

Keep in mind everything under /proc is volatile, so any changes you make are lost after reboot. There are additional internal memory buffers for the TCP Window, allocated per connection:

• /proc/sys/net/ipv4/tcp_rmem - Memory reserved for TCP rcv buffers (reserved memory per connection default)
• /proc/sys/net/ipv4/tcp_wmem - Memory reserved for TCP snd buffers (reserved memory per connection default)

The tcp_rmem and tcp_wmem values contain arrays of three parameter values: the three numbers represent minimum, default, and maximum memory values. Those values are used to bound autotuning and balance memory usage while under global memory stress.

Applying TCP/IP Parameters at System Boot

TCP/IP parameters in Linux are located in /proc/sys/net/ipv4 and /proc/sys/net/core. This is part of the virtual filesystem which resides in system memory (RAM), and any changes to it are volatile, reset when the machine is rebooted.

There are two methods to apply the settings at each reboot.

Method 1: sysctl.conf

Edit /etc/sysctl.conf (or /etc/sysctl.d/sysctl.conf, depending on your distribution). The syntax for setting parameters in this file uses key = value pairs:

net.core.rmem_default = 256960
net.core.rmem_max = 256960
net.core.wmem_default = 256960
net.core.wmem_max = 256960
net.ipv4.tcp_timestamps = 0
net.ipv4.tcp_sack = 0
net.ipv4.tcp_window_scaling = 1

You can see a list of all tweakable parameters by running:

sysctl -a | grep tcp

(or simply sysctl -a for a full list).

Method 2: rc.local

Alternatively, apply the settings at boot time by editing /etc/rc.local, /etc/rc.d/rc.local, or /etc/boot.local depending on your distribution. Note the difference in syntax: you echo the appropriate value into the virtual filesystem.

The parameters above are setting the TCP Window to 256960, disabling timestamps (to avoid 12-byte header overhead), and enabling TCP window scaling:

echo 256960 > /proc/sys/net/core/rmem_default
echo 256960 > /proc/sys/net/core/rmem_max
echo 256960 > /proc/sys/net/core/wmem_default
echo 256960 > /proc/sys/net/core/wmem_max
echo 0 > /proc/sys/net/ipv4/tcp_timestamps
echo 0 > /proc/sys/net/ipv4/tcp_sack
echo 1 > /proc/sys/net/ipv4/tcp_window_scaling

You can change the example values above as desired, depending on your internet connection and maximum bandwidth/latency. There are other parameters you can change from the default if you are confident in what you are doing. Just find the correct path under /proc/sys/net/... and add a line analogous to the others. To revert to the default parameters, comment out or delete the lines from /etc/rc.local and restart.

Changing Current Values

While testing, the current TCP/IP parameters can be edited without a reboot in the following locations:

/proc/sys/net/core/

• rmem_default = Default Receive Window
• rmem_max = Maximum Receive Window
• wmem_default = Default Send Window
• wmem_max = Maximum Send Window

/proc/sys/net/ipv4/

You will find timestamps, window scaling, selective acknowledgements, and more.

Keep in mind the values in /proc will be reset upon reboot. You still need to add the settings in either sysctl.conf or the rc.local syntax described above to have the changes applied at each boot.

To make any new sysctl.conf changes take effect without rebooting:

sysctl -p

To see a list of all relevant tweakable sysctl parameters along with their current values:

sysctl -a | grep tcp

To set a single sysctl value:

sysctl -w variable=value

Example:

sysctl -w net.netfilter.nf_conntrack_tcp_timeout_time_wait=30

TCP Parameters to Consider

TCP_FIN_TIMEOUT

This setting determines the time that must elapse before TCP/IP can release a closed connection and reuse its resources. During the TIME_WAIT state, reopening the connection to the client costs less than establishing a new connection. By reducing this value, TCP/IP can release closed connections faster, making more resources available for new connections. Adjust this in the presence of many connections sitting in the TIME_WAIT state.

sysctl.conf syntax:

net.ipv4.tcp_fin_timeout = 15

(Default: 60 seconds, recommended: 15-30 seconds)

Alternative rc.local syntax:

echo 30 > /proc/sys/net/ipv4/tcp_fin_timeout

TCP_KEEPALIVE_INTERVAL

This determines the wait time between isAlive interval probes.

sysctl.conf syntax:

net.ipv4.tcp_keepalive_intvl = 30

(Default: 75 seconds, recommended: 15-30 seconds)

Alternative rc.local syntax:

echo 30 > /proc/sys/net/ipv4/tcp_keepalive_intvl

TCP_KEEPALIVE_PROBES

This determines the number of probes before timing out.

sysctl.conf syntax:

net.ipv4.tcp_keepalive_probes = 5

(Default: 9, recommended: 5)

Alternative rc.local syntax:

echo 5 > /proc/sys/net/ipv4/tcp_keepalive_probes

TCP_TW_RECYCLE

Enables fast recycling of TIME_WAIT sockets. The default value is 0 (disabled). The sysctl documentation incorrectly states the default as enabled. It can be changed to 1 (enabled) in many cases. Known to cause issues with host-based load balancing and failover if enabled, so use with caution.

sysctl.conf syntax:

net.ipv4.tcp_tw_recycle = 1

(Boolean, default: 0)

Alternative rc.local syntax:

echo 1 > /proc/sys/net/ipv4/tcp_tw_recycle

TCP_TW_REUSE

Allows reusing sockets in TIME_WAIT state for new connections when it is safe from a protocol viewpoint. Default value is 0 (disabled). It is generally a safer alternative to tcp_tw_recycle.

sysctl.conf syntax:

net.ipv4.tcp_tw_reuse = 1

(Boolean, default: 0)

Alternative rc.local syntax:

echo 1 > /proc/sys/net/ipv4/tcp_tw_reuse

Linux Netfilter Tweaks

To list netfilter parameters:

sysctl -a | grep netfilter

Add the following to /etc/sysctl.conf to tune individual parameters:

# Reduce TIME_WAIT from the 120s default to 30s
net.netfilter.nf_conntrack_tcp_timeout_time_wait = 30

# Reduce FIN_WAIT from the 120s default to 30s
net.netfilter.nf_conntrack_tcp_timeout_fin_wait = 30

Commit the sysctl.conf changes without rebooting (and test for possible syntax errors):

sysctl -p

To check sysctl parameters:

sysctl -a

To check the actual number of current connections in the TIME_WAIT state:

netstat -n | grep TIME_WAIT | wc -l

Kernel Recompile Option

There is another method to directly set the default TCP/IP parameters involving a kernel recompile. If you are brave enough, look for the parameters in the following files:

• /LINUX-SOURCE-DIR/include/linux/skbuff.h (look for SK_WMEM_MAX and SK_RMEM_MAX)
• /LINUX-SOURCE-DIR/include/net/tcp.h (look for MAX_WINDOW and MIN_WINDOW)

Determine Connection States

It is often useful to decrease some of the TCP timeouts to release resources faster and reduce memory use. The default TCP timeouts may leave too many connections in the TIME_WAIT state. To see a list of all current connections to the machine and their states:

netstat -tan | grep ':80 ' | awk '{print $6}' | sort | uniq -c

You will be presented with output similar to the following:

   4 CLOSING
  12 ESTABLISHED
   4 FIN_WAIT1
  14 FIN_WAIT2
  12 LAST_ACK
   1 LISTEN
  10 SYN_RECV
 273 TIME_WAIT

This information is useful for determining whether you need to tweak some of the timeouts above.

SYN Flood Protection

The following settings added to sysctl.conf make a server more resistant to SYN flood attacks. They configure the kernel to use the SYN cookies mechanism with a backlog queue of 1024 connections, and set the SYN and SYN/ACK retries to an effective ceiling of about 45 seconds. The defaults vary depending on kernel version and distribution, so check them with:

sysctl -a | grep syn

net.ipv4.tcp_max_syn_backlog = 1024
net.ipv4.tcp_syn_retries = 6
net.ipv4.tcp_synack_retries = 3
net.ipv4.tcp_syncookies = 1

TCP Congestion Control Algorithm

By default, the Linux kernel uses the CUBIC congestion control/avoidance algorithm, which is very good and probably the most popular. A newer alternative is BBR (version 2). BBR was first introduced in kernel 4.9, with a patch/improvement to its functionality as recently as November 2020 in kernel 5.9.11.

To see your current kernel version:

uname -r

To view a list of available congestion control algorithms:

sysctl net.ipv4.tcp_available_congestion_control

To see your current congestion control algorithm:

sysctl net.ipv4.tcp_congestion_control

Note that only CUBIC and RENO are available by default. To switch to BBR on kernels 4.9 and newer, edit /etc/sysctl.conf and add the following two lines:

net.core.default_qdisc = fq
net.ipv4.tcp_congestion_control = bbr

Then reload sysctl:

sysctl -p

Verify the new congestion control algorithm is in use:

sysctl net.ipv4.tcp_congestion_control

References

• TCP Variables (ipsysctl-tutorial)
• ip-sysctl parameters reference

• Networking
• Technology

• SysAdmin
• Bash