Mod Rewrite and Blocking Bad Bot

Luke Hebert

- 2 minutes read - 278 words

If a particular bot is hammering your Apache server and you need a fast, no-restart way to shut it down, mod_rewrite gives you that in a few lines. The snippet below matches any User-Agent that starts with “bot” (case-insensitive) and returns a 403 Forbidden.

The Rewrite Rules

Add the following to your .htaccess file (or a virtual-host <Directory> block):

RewriteEngine on
Options +FollowSymlinks
RewriteBase /

RewriteCond %{HTTP_USER_AGENT} ^bot* [NC]
RewriteRule .* - [F,L]

What Each Line Does

  • RewriteEngine on enables the rewrite engine for the current directory.
  • Options +FollowSymlinks is a prerequisite for mod_rewrite to function.
  • RewriteBase / sets the base path for relative rewrite rules.
  • RewriteCond %{HTTP_USER_AGENT} ^bot* [NC] checks the User-Agent header against the pattern ^bot* case-insensitively.
  • RewriteRule .* - [F,L] matches every request, returns a 403 Forbidden, and stops further rule processing.

Things to Keep in Mind

Warning

The pattern ^bot* matches any User-Agent that begins with “bot” followed by zero or more “t” characters. That means it will catch “bot”, “botbot”, “botbotbot”, and so on, but it will not catch “botnet” or “botcrawler” because those have a non-“t” character after “bot”. If you want to block everything that starts with “bot”, use ^bot instead.

Note

Legitimate crawlers (Googlebot, Bingbot, etc.) also start with “bot”. If you need to allow specific crawlers, add a preceding RewriteCond that excludes them, for example:

RewriteCond %{HTTP_USER_AGENT} !^Googlebot [NC]
RewriteCond %{HTTP_USER_AGENT} !^Bingbot [NC]
RewriteCond %{HTTP_USER_AGENT} ^bot [NC]
RewriteRule .* - [F,L]

Verifying the Block

After saving the file, test with curl:

curl -I -A "badbot" https://example.com/

You should see a 403 Forbidden response. A normal browser User-Agent should still return 200 OK.

References