# Storage backends

BotStopper needs to store temporary data in order to determine if a user is legitimate or not. Administrators should choose a storage backend based on their infrastructure needs. Each backend has its own advantages and disadvantages.

BotStopper offers the following storage backends:

- [`memory`](#memory) -- A simple in-memory hashmap
- [`bbolt`](#bbolt) -- An on-disk key/value store backed by [bbolt](https://github.com/etcd-io/bbolt), an embedded key/value database for Go programs
- [`s3api`](#s3api) -- A network-backed storage layer backed by [object storage](https://en.wikipedia.org/wiki/Object_storage), specifically using the [S3 API](https://docs.aws.amazon.com/AmazonS3/latest/API/Type_API_Reference.html).
- [`valkey`](#valkey) -- A remote in-memory key/value database backed by [Valkey](https://valkey.io/) (or another database compatible with the [RESP](https://redis.io/docs/latest/develop/reference/protocol-spec/) protocol)

If no storage backend is set in the policy file, BotStopper will use the [`memory`](#memory) backend by default. This is equivalent to the following in the policy file:

```yaml
store:
  backend: memory
  parameters: {}
```

### `memory`

The memory backend is an in-memory cache. This backend works best if you don't use multiple instances of BotStopper or don't have mutable storage in the environment you're running BotStopper in.

| Should I use this backend?                                    | Yes/no |
| :------------------------------------------------------------ | :----- |
| Are you running only one instance of BotStopper for this service? | ✅ Yes |
| Does your service get a lot of traffic?                       | 🚫 No  |
| Do you want to store data persistently when BotStopper restarts?  | 🚫 No  |
| Do you run BotStopper without mutable filesystem storage?         | ✅ Yes |

The biggest downside is that there is not currently a limit to how much data can be stored in memory. This will be addressed at a later time.

> **Warning.**
> The in-memory backend exists mostly for validation, testing, and to ensure that the default configuration of BotStopper works as expected. Do not use this persistently in production.

#### Configuration

The memory backend does not require any configuration to use.

### `bbolt`

An on-disk storage layer powered by [bbolt](https://github.com/etcd-io/bbolt), a high performance embedded key/value database used by containerd, etcd, Kubernetes, and NATS. This backend works best if you're running BotStopper on a single host and get a lot of traffic.

| Should I use this backend?                                    | Yes/no |
| :------------------------------------------------------------ | :----- |
| Are you running only one instance of BotStopper for this service? | ✅ Yes |
| Does your service get a lot of traffic?                       | ✅ Yes |
| Do you want to store data persistently when BotStopper restarts?  | ✅ Yes |
| Do you run BotStopper without mutable filesystem storage?         | 🚫 No  |

When BotStopper opens a bbolt database, it takes an exclusive lock on that database. Other instances of BotStopper or other tools cannot view the bbolt database while it is locked by another instance of BotStopper. If you run multiple instances of BotStopper for different services, give each its own `bbolt` configuration.

#### Configuration

The `bbolt` backend takes the following configuration options:

| Name   | Type | Example            | Description                                                                                                                  |
| :----- | :--- | :----------------- | :--------------------------------------------------------------------------------------------------------------------------- |
| `path` | path | `/data/anubis.bdb` | The filesystem path for the BotStopper bbolt database. BotStopper requires write access to the folder containing the bbolt database. |

Example:

If you have persistent storage mounted to `/data`, then your store configuration could look like this:

```yaml
store:
  backend: bbolt
  parameters:
    path: /data/anubis.bdb
```

### `s3api`

A network-backed storage layer backed by [object storage](https://en.wikipedia.org/wiki/Object_storage), specifically using the [S3 API](https://docs.aws.amazon.com/AmazonS3/latest/API/Type_API_Reference.html). This can be backed by any S3-compatible object storage service such as:

- [AWS S3](https://aws.amazon.com/s3/)
- [Cloudflare R2](https://www.cloudflare.com/developer-platform/products/r2/)
- [Hetzner Object Storage](https://www.hetzner.com/storage/object-storage/)
- [Minio](https://www.min.io/)
- [Tigris](https://www.tigrisdata.com/)

If you are using a cloud platform, they likely provide an S3 compatible object storage service. If not, you may want to choose [one of the fastest options](https://www.tigrisdata.com/blog/benchmark-small-objects/).

| Should I use this backend?                                    | Yes/no |
| :------------------------------------------------------------ | :----- |
| Are you running only one instance of BotStopper for this service? | 🚫 No  |
| Does your service get a lot of traffic?                       | ✅ Yes |
| Do you want to store data persistently when BotStopper restarts?  | ✅ Yes |
| Do you run BotStopper without mutable filesystem storage?         | ✅ Yes |

> **Note.**
> Using this backend will cause a lot of S3 operations, at least one for creating challenges, one for invalidating challenges, one for updating challenges to prevent double-spends, and one for removing challenges.

#### Configuration

The `s3api` backend takes the following configuration options:

| Name         | Type    | Example       | Description                                                                                                                                 |
| :----------- | :------ | :------------ | :------------------------------------------------------------------------------------------------------------------------------------------ |
| `bucketName` | string  | `anubis-data` | (Required) The name of the dedicated bucket for BotStopper to store information in.                                                             |
| `pathStyle`  | boolean | `false`       | If true, use path-style S3 API operations. Please consult your storage provider's documentation if you don't know what you should put here. |

> **Note.**
> You should probably enable a lifecycle expiration rule for buckets containing BotStopper data. Here is an example policy:
>
> ```json
> {
>   "Rules": [
>     {
>       "Status": "Enabled",
>       "Expiration": {
>         "Days": 7
>       }
>     }
>   ]
> }
> ```
>
> Adjust this as facts and circumstances demand, but 7 days should be enough for anyone.

Example:

Assuming your environment looks like this:

```sh
# All of the following are fake credentials that look like real ones.
AWS_ACCESS_KEY_ID=accordingToAllKnownRulesOfAviation
AWS_SECRET_ACCESS_KEY=thereIsNoWayABeeShouldBeAbleToFly
AWS_REGION=yow
AWS_ENDPOINT_URL_S3=https://yow.s3.probably-not-malware.lol
```

Then your configuration would look like this:

```yaml
store:
  backend: s3api
  parameters:
    bucketName: techaro-prod-anubis
    pathStyle: false
```

### `valkey`

[Valkey](https://valkey.io/) is an in-memory key/value store that clients access over the network. This allows multiple instances of BotStopper to share information and does not require each instance of BotStopper to have persistent filesystem storage.

> **Note.**
> You can also use [Redis™](http://redis.io/) with BotStopper.

This backend is ideal if you are running multiple instances of BotStopper in a worker pool (eg: Kubernetes Deployments with a copy of BotStopper in each Pod).

| Should I use this backend?                                    | Yes/no |
| :------------------------------------------------------------ | :----- |
| Are you running only one instance of BotStopper for this service? | 🚫 No  |
| Does your service get a lot of traffic?                       | ✅ Yes |
| Do you want to store data persistently when BotStopper restarts?  | ✅ Yes |
| Do you run BotStopper without mutable filesystem storage?         | ✅ Yes |
| Do you have Redis™ or Valkey installed?                       | ✅ Yes |

#### Configuration

The `valkey` backend takes the following configuration options:

| Name       | Type   | Example                 | Description                                                                                                                                       |
| :--------- | :----- | :---------------------- | :------------------------------------------------------------------------------------------------------------------------------------------------ |
| `cluster`  | bool   | `false`                 | If true, use [Redis™ Clustering](https://redis.io/topics/cluster-spec) for storing BotStopper data.                                                   |
| `sentinel` | object | `{}`                    | See [Redis™ Sentinel docs](#redis-sentinel) for more detail and examples                                                                          |
| `url`      | string | `redis://valkey:6379/0` | The URL for the instance of Redis™ or Valkey that BotStopper should store data in. This is in the same format as `REDIS_URL` in many cloud providers. |

Example:

If you have an instance of Valkey running with the hostname `valkey.int.techaro.lol`, then your store configuration could look like this:

```yaml
store:
  backend: valkey
  parameters:
    url: "redis://valkey.int.techaro.lol:6379/0"
```

This would have the Valkey client connect to host `valkey.int.techaro.lol` on port `6379` with database `0` (the default database).

#### Redis™ Sentinel

If you are using [Redis™ Sentinel](https://redis.io/docs/latest/operate/oss_and_stack/management/sentinel/) for a high availability setup, you need to configure the `sentinel` object. This object takes the following configuration options:

| Name         | Type                     | Example               | Description                                                                                                                                               |
| :----------- | :----------------------- | :-------------------- | :-------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `addr`       | string or list of string | `10.43.208.130:26379` | (Required) The host and port of the Redis™ Sentinel server. When possible, use DNS names for this. If you have multiple addresses, supply a list of them. |
| `clientName` | string                   | `Anubis`              | The client name reported to Redis™ Sentinel. Set this if you want to track BotStopper connections to your Redis™ Sentinel.                                    |
| `masterName` | string                   | `mymaster`            | (Required) The name of the master in the Redis™ Sentinel configuration. This is used to discover where to find client connection hosts/ports.             |
| `username`   | string                   | `azurediamond`        | The username used to authenticate against the Redis™ Sentinel and Redis™ servers.                                                                         |
| `password`   | string                   | `REPLACE_WITH_A_RANDOM_PASSWORD`             | The password used to authenticate against the Redis™ Sentinel and Redis™ servers.                                                                         |

See [policy configuration](/docs/policies.md) for related topics.