BotStopper has the ability to let you import snippets of configuration into the main configuration file. This allows you to break up your config into smaller parts that get logically assembled into one big file.
EG:
bots: # Pathological bots to deny - # This correlates to data/bots/ai-catchall.yaml in the source tree import: (data)/bots/ai-catchall.yaml - import: (data)/bots/cloudflare-workers.yaml # Import all the rules in the default configuration - import: (data)/meta/default-config.yaml
Note.
The (data) pseudo-folder corresponds to a copy of the data as of the time BotStopper was compiled. If you want to dump a copy of the (data) folder out of your BotStopper binary, run this command:
anubis --extract-resources ./var/anubis-resources
This will extract the contents of the (data) folder compiled into your copy of BotStopper as well as the contents of static filesystem used for web assets. For example:
config is not valid:config.BotOrImport: rule definition is invalid, you must set either bot rules or an import statement, not both
Paths can either be prefixed with (data) to import from the the data folder in the BotStopper source tree or anywhere on the filesystem. If you don't have access to the BotStopper source tree, check /usr/share/docs/anubis/data or in the tarball you extracted BotStopper from.
Importing many files at once
If you want to import policies from many files at once, the import path can contain wildcards. Wildcard matches let you add rules by dropping a file into a folder instead of editing your main configuration file every time.
For example:
bots: # Every rule file in the (data)/crawlers folder. - import: (data)/crawlers/*.yaml # Every rule file you put in /etc/techaro-botstopper/known-good. - import: /etc/techaro-botstopper/known-good/*.yaml
Supported wildcards
BotStopper uses the Go standard library function filepath.Match for matching. It supports four wildcards:
Wildcard
Matches
Example
*
Any run of characters in one folder
crawlers/*.yaml matches crawlers/kagibot.yaml, but not crawlers/eu/kagibot.yaml.
?
Exactly one character
??-allow.yaml matches 00-allow.yaml and xe-allow.yaml, but not 1-allow.yaml.
[0-9]
One character from a set or range
[0-9][0-9]-*.yaml matches 00-monitoring.yaml, but not xe-monitoring.yaml.
BotStopper matches requests top to bottom, similar to Apache's DENY, ALLOW ACL. BotStopper uses the first rule that matches the request exactly. A rule that loads early can shadow a rule that loads later.
When BotStopper loads files imported with a wildcard match, it sorts the files in natural sort order. To control the order rules are loaded, put a number at the beginning of each filename:
Natural sort compares numbers numerically and words alphabetically. This makes 02-deny.yaml load before 10-deny.yaml.
Tip.
Make sure your indentation is consistent across all files matched in a single import statement. Format your YAML files with yq or configure your editor to format YAML files when you save.
Errors from wildcard imports
A wildcard import that fails stops BotStopper at startup. These are the common causes:
Error
Cause
glob pattern matched no files
The pattern matched nothing. Check the folder, the file extension, and the syntax.
can't parse (data)/*: read apps/: is a directory
The pattern matched a folder. End the pattern with .yaml so that it only matches files.
can't parse ...: yaml: line N: ...
One of the matched files is not a top level list, or its indentation is wrong.
Patterns that match no files result in errors. The message names the pattern that failed:
glob pattern matched no files: "(data)/bots/*.yml", note that recursive wildcards ("**") and alternation ("{yaml,yml}") are not supported
Importing the default configuration
If you want to base your configuration off of the default configuration, import (data)/meta/default-config.yaml:
bots: - import: (data)/meta/default-config.yaml # Write your rules here
This will keep your configuration up to date as BotStopper adapts to emerging threats.
How do I exempt most modern browsers from BotStopper challenges?
If you want to exempt most modern browsers from BotStopper challenges, import (data)/common/acts-like-browser.yaml:
bots: - import: (data)/meta/default-config.yaml - import: (data)/common/acts-like-browser.yaml # Write your rules here
These rules will allow traffic that "looks like" it's from a modern copy of Edge, Safari, Chrome, or Firefox. These rules used to be enabled by default, however user reports have suggested that AI scraper bots have adapted to conform to these rules to scrape without regard for the infrastructure they are attacking.
Use these rules at your own risk.
Importing from imports
You can also import from an imported file in case you want to import an entire folder of rules at once.
Use this with care, you can easily get yourself into a state where BotStopper recursively imports things for eternity if you are not careful.
If you need to import many files, please consider using glob matching instead.
Writing snippets
Snippets can be written in either JSON or YAML, with a preference for YAML. When writing a snippet, write the bot rules you want directly at the top level of the file in a list.
