IPv4 Kernel Stack Optimizations

Luke Hebert

- 4 minutes read - 740 words

Altering the TCP/IP stack in the Linux kernel can be complex, but it pays off when the default congestion algorithms cannot keep up with your workload. The settings below come from years of work in web hosting and high-performance compute environments. Keep in mind that many other controls exist for performance and security, and some of these changes may affect slower clients or those behind high-latency links.

Important

The content here is accurate at the time of posting but things in the Linux Kernel change rapidly. Make sure you verify items you see here against your current release.

Step 1: Identify the Active Congestion Control Algorithm

Log on to the system and check which algorithm is currently in use:

# cat /proc/sys/net/ipv4/tcp_congestion_control

Make a note of the result.

Step 2: Check Available Algorithms

# cat /proc/sys/net/ipv4/tcp_available_congestion_control
Note

Even on systems with pre-compiled kernels, not all modules may be loaded. Some modules need to be loaded explicitly based on your algorithm preference. BBR, written by Google and used on some of their platforms, is only available on relatively new kernels (not the 3.x branch).

Step 3: Find Unloaded Modules

If the algorithm you want is not listed, the module is likely unloaded from the kernel. You can inspect what is available on disk:

# ls -la /lib/modules/$(uname -r)/kernel/net/ipv4
Note

Not every item prefixed with tcp_ is a congestion control algorithm. A non-exhaustive list is available on Wikipedia or other reference resources.

Step 4: Load the Desired Module

Once you identify the module, load it with modprobe. Substitute the module name as needed:

# modprobe tcp_yeah

Step 5: Write the Sysctl Configuration

Open a text editor such as vim or nano and create a file at /etc/sysctl.d/99-congest.conf (or a similar name following your naming convention). Paste the content below:

Important

This configuration is predefined to use the yeah algorithm, but you can swap in any algorithm you prefer. Be aware that some algorithms do not need or support SACK/FACK; they use a different rapid-recovery mechanism such as RACK. It is also possible to combine multiple algorithms. Use your judgment based on your use case and understanding of these options.

### Increase socket input/output queue memory (at the cost of more CPU usage)

# Maximum Socket Send Buffer
net.core.wmem_max = 12582912

# Maximum Socket Receive Buffer
net.core.rmem_max = 12582912

# Increase the read-buffer space allocatable
net.ipv4.tcp_rmem = 10240 87380 12582912

# Increase the write-buffer space allocatable
net.ipv4.tcp_wmem = 10240 87380 12582912

### Adjust incoming connections and connection backlog

# Set the maximum number of allowed incoming connections
net.core.somaxconn = 10240

# Set the maximum network backlog. If the backlog argument is greater than
# /proc/sys/net/core/somaxconn, it is silently truncated to that value.
# Default is 128. In kernels before 2.4.25 this was a hard-coded SOMAXCONN of 128.
net.core.netdev_max_backlog = 4000

# Set Max SYN Packet Backlog
net.ipv4.tcp_max_syn_backlog = 10240

# Adjust the Local Port Range and Sockets Available
net.ipv4.ip_local_port_range = 2000 65535

### Adjust Wait Timeouts

# Decrease the default tcp_fin_timeout
net.ipv4.tcp_fin_timeout = 15

# Decrease the default tcp_keepalive_time (default is 7200)
net.ipv4.tcp_keepalive_time = 180
net.ipv4.tcp_keepalive_probes = 5
net.ipv4.tcp_keepalive_intvl = 15

### Adjust Congestion Control

# Turn on TCP window scaling
net.ipv4.tcp_window_scaling = 1

# Turn on SACK (Selective Acknowledgements improve congestion control)
net.ipv4.tcp_sack = 1

# Turn on FACK (Forward Acknowledgements improve congestion control)
net.ipv4.tcp_fack = 1

# Disable ICMP Redirect Acceptance
net.ipv4.conf.all.accept_redirects = 0

# Don't log spoofed packets, source-routed packets, or redirect packets
net.ipv4.conf.all.log_martians = 0

# Re-use sockets in TIME-WAIT state
net.ipv4.tcp_tw_reuse = 1

# Increase the TIME-WAIT bucket pool size to mitigate simple DOS attacks
net.ipv4.tcp_max_tw_buckets = 1440000

# Set congestion control algorithm (typically cubic or Reno)
# Note: NewReno/Reno may have trouble with SACK/FACK.
#
# Recommend westwood on moderate-sized uplinks and YeaH on high-speed uplinks.
#
# Your mileage may vary depending on what is available in your OS.
#
# Wikipedia is not exhaustive; there is a large volume of information online
# about different algorithms.
# https://en.wikipedia.org/wiki/TCP_congestion_control
#
# RFC5681
# http://www.rfc-editor.org/rfc/pdfrfc/rfc5681.txt.pdf
net.ipv4.tcp_congestion_control = yeah

Step 6: Apply the Changes

To make the changes take effect immediately without a reboot, force the system to reload its running stack parameters:

# sysctl -f /etc/sysctl.d/99-congest.conf

You can review all current stack settings with a command like the one below. The previous command will also report any settings that failed to apply.

# sysctl -a | grep net | less