BOTSTOPPER FROM TECHAROPRODUCT DOCUMENTATION

Bot rules

Bot policies let you customize the rules that BotStopper uses to allow, deny, or challenge incoming requests. Currently you can set policies by the following matches:

Request path
User agent string
HTTP request header values
Importing other configuration snippets

Policy files can use JSON or YAML.

Here's an example rule that denies Amazonbot:

- name: amazonbot
  user_agent_regex: Amazonbot
  action: DENY

When this rule is evaluated, BotStopper will check the User-Agent string of the request. If it contains Amazonbot, BotStopper will send an error page to the user saying that access is denied, but in such a way that makes scrapers think they have correctly loaded the webpage.

This minimal policy shows path exceptions and a browser challenge:

bots:
  - name: cloudflare-workers
    headers_regex:
      CF-Worker: .*
    action: DENY
  - name: well-known
    path_regex: ^/.well-known/.*$
    action: ALLOW
  - name: favicon
    path_regex: ^/favicon.ico$
    action: ALLOW
  - name: robots-txt
    path_regex: ^/robots.txt$
    action: ALLOW
  - name: generic-browser
    user_agent_regex: Mozilla
    action: CHALLENGE

This allows requests to /.well-known, /favicon.ico, /robots.txt, and challenges any request that has the word Mozilla in its User-Agent string. The default policy file is a bit more cohesive, but this should be more than enough for most users.

If no terminal bot rule matches, BotStopper evaluates the request weight against thresholds. If no threshold matches, it allows the request. For more details on this default behavior and its implications, see Default allow behavior.

Writing your own rules

There are four actions that can be returned from a rule:

Action	Effects
`ALLOW`	Bypass all further checks and send the request to the backend.
`DENY`	Deny the request and send back an error message that scrapers think is a success.
`CHALLENGE`	Show a challenge page and/or validate that clients have passed a challenge.
`WEIGH`	Change the request weight for this request. See the request weight docs for more information.

Name your rules in lower case using kebab-case. Rule names will be exposed in Prometheus metrics.

Challenge configuration

Rules can also have their own challenge settings. These are customized using the "challenge" key. For example, here is a rule that makes challenges artificially hard for connections with the substring "bot" in their user agent:

This rule has been known to have a high false positive rate in testing. Please use this with care.

# Punish any bot with "bot" in the user-agent string
- name: generic-bot-catchall
  user_agent_regex: (?i:bot|crawler)
  action: CHALLENGE
  challenge:
    difficulty: 6
    algorithm: fast

Challenges can be configured with these settings:

Key	Example	Description
`difficulty`	`4`	The challenge difficulty (number of leading zeros) for proof-of-work. See Why does BotStopper use Proof-of-Work? for more details.
`algorithm`	`"fast"`	The challenge method to use. See the list of challenge methods for more information.

Remote IP based filtering

The remote_addresses field of a Bot rule allows you to set the IP range that this ruleset applies to.

For example, you can allow a search engine to connect if and only if its IP address matches the ones they published:

- name: qwantbot
  user_agent_regex: \+https\://help\.qwant\.com/bot/
  action: ALLOW
  # https://help.qwant.com/wp-content/uploads/sites/2/2025/01/qwantbot.json
  remote_addresses: ["91.242.162.0/24"]

This also works at an IP range level without any other checks:

name: internal-network
action: ALLOW
remote_addresses:
  - 100.64.0.0/10

See policy configuration for related topics.

PREVIOUS← Request weight NEXTStorage backends →

Loading documentation...

The document will appear here shortly.