Check Web Crawler Impact

If your site is getting hammered by search engine bots and you need a fast way to see who is doing the crawling, this little bash function does the job. It scans your Apache domain logs for common crawler user agents and ranks them by hit count, so you can immediately spot whether a particular bot is behaving normally or flooding your server.

The Function

crawlers() {
  domain=$1
  awk '/[Bb]ot|[Gg]oogle|[Yy]ahoo|[Ss]lurp|[Bb]ing|[Mm][Ss][Nn]|[Yy]andex|[Bb]aidu/ {print $1}' \
    /usr/local/apache/domlogs/$domain | \
    sort | uniq -c | sort -nk1 | tail
}

Usage

Call it with the domain you want to inspect:

crawlers example.com

The output lists each crawler IP or user-agent string followed by the number of hits, sorted from highest to lowest. The tail at the end shows the top entries.

What to Look For

• Sudden spikes in a single bot’s hit count compared to your normal baseline.
• Unfamiliar user-agent strings that match the regex but are not a bot you expect to see.
• A single IP generating the bulk of crawler traffic, which can indicate a misconfigured proxy or a scraping operation.

Notes

Where to Put It

Drop the function into your ~/.bashrc or a shared script file so it is available on any host where you manage Apache logs. No additional dependencies beyond awk, sort, uniq, and tail are needed, all of which ship with standard Linux distributions.

• SysAdmin

• Bash
• Troubleshooting