# Bot rules

Bot policies let you customize the rules that BotStopper uses to allow, deny, or challenge incoming requests. Currently you can set policies by the following matches:

- Request path
- User agent string
- HTTP request header values
- [Importing other configuration snippets](/docs/configuration/import.md)

Policy files can use JSON or YAML.

Here's an example rule that denies [Amazonbot](https://developer.amazon.com/en/amazonbot):

```yaml
- name: amazonbot
  user_agent_regex: Amazonbot
  action: DENY
```

When this rule is evaluated, BotStopper will check the `User-Agent` string of the request. If it contains `Amazonbot`, BotStopper will send an error page to the user saying that access is denied, but in such a way that makes scrapers think they have correctly loaded the webpage.

This minimal policy shows path exceptions and a browser challenge:

```yaml
bots:
  - name: cloudflare-workers
    headers_regex:
      CF-Worker: .*
    action: DENY
  - name: well-known
    path_regex: ^/.well-known/.*$
    action: ALLOW
  - name: favicon
    path_regex: ^/favicon.ico$
    action: ALLOW
  - name: robots-txt
    path_regex: ^/robots.txt$
    action: ALLOW
  - name: generic-browser
    user_agent_regex: Mozilla
    action: CHALLENGE
```

This allows requests to [`/.well-known`](https://en.wikipedia.org/wiki/Well-known_URI), `/favicon.ico`, `/robots.txt`, and challenges any request that has the word `Mozilla` in its User-Agent string. The [default policy file](https://github.com/TecharoHQ/anubis/blob/main/data/botPolicies.yaml) is a bit more cohesive, but this should be more than enough for most users.

If no terminal bot rule matches, BotStopper evaluates the request weight against thresholds. If no threshold matches, it allows the request. For more details on this default behavior and its implications, see [Default allow behavior](/docs/configuration/default-allow-behavior.md).

### Writing your own rules

There are four actions that can be returned from a rule:

| Action      | Effects                                                                                                                             |
| :---------- | :---------------------------------------------------------------------------------------------------------------------------------- |
| `ALLOW`     | Bypass all further checks and send the request to the backend.                                                                      |
| `DENY`      | Deny the request and send back an error message that scrapers think is a success.                                                   |
| `CHALLENGE` | Show a challenge page and/or validate that clients have passed a challenge.                                                         |
| `WEIGH`     | Change the [request weight](/docs/policies/request-weight.md) for this request. See the [request weight](/docs/policies/request-weight.md) docs for more information. |

Name your rules in lower case using kebab-case. Rule names will be exposed in Prometheus metrics.

### Challenge configuration

Rules can also have their own challenge settings. These are customized using the `"challenge"` key. For example, here is a rule that makes challenges artificially hard for connections with the substring "bot" in their user agent:

This rule has been known to have a high false positive rate in testing. Please use this with care.

```yaml
# Punish any bot with "bot" in the user-agent string
- name: generic-bot-catchall
  user_agent_regex: (?i:bot|crawler)
  action: CHALLENGE
  challenge:
    difficulty: 6
    algorithm: fast
```

Challenges can be configured with these settings:

| Key          | Example  | Description                                                                                                                                                      |
| :----------- | :------- | :--------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `difficulty` | `4`      | The challenge difficulty (number of leading zeros) for proof-of-work. See [Why does BotStopper use Proof-of-Work?](https://anubis.techaro.lol/docs/design/why-proof-of-work) for more details. |
| `algorithm`  | `"fast"` | The challenge method to use. See [the list of challenge methods](/docs/configuration.md#challenge-methods) for more information.                                              |

### Remote IP based filtering

The `remote_addresses` field of a Bot rule allows you to set the IP range that this ruleset applies to.

For example, you can allow a search engine to connect if and only if its IP address matches the ones they published:

```yaml
- name: qwantbot
  user_agent_regex: \+https\://help\.qwant\.com/bot/
  action: ALLOW
  # https://help.qwant.com/wp-content/uploads/sites/2/2025/01/qwantbot.json
  remote_addresses: ["91.242.162.0/24"]
```

This also works at an IP range level without any other checks:

```yaml
name: internal-network
action: ALLOW
remote_addresses:
  - 100.64.0.0/10
```

See [policy configuration](/docs/policies.md) for related topics.