IP Tables Rate Limit Bad Bots

Luke Hebert

- 2 minutes read - 400 words

Bad bots can flood your web server with requests, eat bandwidth, and slow down legitimate traffic. A quick way to rein them in is to use iptables to match the bot’s user-agent string and drop any connection that exceeds a small request threshold within a short time window.

How It Works

Each rule below does three things:

  1. Matches TCP traffic on port 80.
  2. Looks for a specific bot string in the packet payload using the string match with the Boyer-Moore algorithm.
  3. Uses the recent module to track hits. If the same source IP exceeds the hit count within the given number of seconds, the packet is dropped.

Rules

iptables -I INPUT -p tcp --dport 80 -m string --algo bm --string "bingbot" -m recent --rcheck --seconds 60 --hitcount 5 -j DROP
iptables -I INPUT -p tcp --dport 80 -m string --algo bm --string "msnbot" -m recent --rcheck --seconds 60 --hitcount 5 -j DROP
iptables -I INPUT -p tcp --dport 80 -m string --algo bm --string "googlebot" -m recent --rcheck --seconds 60 --hitcount 5 -j DROP
iptables -I INPUT -p tcp --dport 80 -m string --algo bm --string "yahoo" -m recent --rcheck --seconds 60 --hitcount 5 -j DROP
iptables -I INPUT -p tcp --dport 80 -m string --algo bm --string "yahoo! slurp" -m recent --rcheck --seconds 60 --hitcount 5 -j DROP

Tuning

  • --seconds 60 sets the time window. Lower it (for example, 10) to be more aggressive.
  • --hitcount 5 is the number of requests allowed before the drop kicks in. Raise it if legitimate crawlers are getting caught.
  • Add or remove bot strings to match the traffic you actually see. Check your access logs for the user-agent values that are causing the most noise.
Warning

The string match inspects packet payloads, which adds CPU overhead on busy servers. Test the rules in a staging environment or during a low-traffic window before deploying them to production.

Note

These rules are inserted at the top of the INPUT chain (-I). If you already have other INPUT rules, verify the ordering with iptables -L INPUT -n -v to make sure the bot rules fire before any ACCEPT rules that would let the traffic through.

Persisting the Rules

On most distributions you can save the current ruleset so it survives a reboot:

iptables-save > /etc/iptables/rules.v4

Make sure the iptables-persistent package (Debian/Ubuntu) or iptables-services (RHEL/CentOS) is installed so the saved file is loaded at boot.