In-memory store

In-memory data stores (such as Redis) are high-performance databases that store data entirely in memory, enabling extremely fast read and write operations. These systems are often used for caching, message brokering, and as temporary storage for dynamic data that does not require long-term persistence.

In Stalwart, in-memory stores play a crucial role in handling a wide variety of tasks. They are used for storing:

• Rate limiting and fail2ban data, such as the number of messages sent by a specific sender or the number of failed authentication attempts from a specific IP address.
• Distributed locks for managing concurrent tasks, such as purging accounts, processing email queues, and running housekeeping tasks.
• OAuth authorization codes to validate the authorization process.
• ACME tokens for SSL/TLS certificate management.
• Certain e-mail tracking data, such as greylisting tokens and Sieve auto-responder IDs.

Backends

Stalwart supports the following backends as in-memory stores:

• Redis, Memcached, or compatible: Ideal for high-performance, distributed, or heavy-load environments. Redis, in particular, is recommended for such cases because of its speed and versatility as a cache, database, and message broker.
• Data store: For administrators aiming to minimize external dependencies, any supported data store backend (SQL, FoundationDB, RocksDB) can also function as an in-memory store. While data store backends provide flexibility, Redis is better suited for heavy loads and distributed setups due to its optimized performance.

Key Prefixes

In-memory stores operate as key-value stores, where each key is prefixed with a unique identifier to prevent conflicts. Stalwart uses predefined prefixes for different types of data, ensuring clear organization within the store.

Below is a mapping of key prefixes used by Stalwart, including their assigned unsigned integer values:

Short name	Description	Integer prefix
KV_ACME	ACME tokens for SSL/TLS certificate management	0
KV_OAUTH	OAuth tokens for authentication	1
KV_RATE_LIMIT_RCPT	Rate limiting data for recipient addresses	2
KV_RATE_LIMIT_SCAN	Rate limiting data for email scanning	3
KV_RATE_LIMIT_LOITER	Rate limiting data for idle connections	4
KV_RATE_LIMIT_AUTH	Rate limiting data for authentication attempts	5
KV_RATE_LIMIT_SMTP	Rate limiting data for SMTP throttles	6
KV_RATE_LIMIT_CONTACT	Rate limiting data for contact forms	7
KV_RATE_LIMIT_HTTP_AUTHENTICATED	Rate limiting data for authenticated HTTP requests	8
KV_RATE_LIMIT_HTTP_ANONYMOUS	Rate limiting data for anonymous HTTP requests	9
KV_RATE_LIMIT_IMAP	Rate limiting data for IMAP connections	10
KV_TOKEN_REVISION	Revision number for access tokens	11
KV_GREYLIST	Spam filter greylist tokens	16
KV_TRUSTED_REPLY	Spam filter trusted replies tracking	19
KV_LOCK_PURGE_ACCOUNT	Lock for purging accounts	20
KV_LOCK_QUEUE_MESSAGE	Lock for SMTP message queues	21
KV_LOCK_QUEUE_REPORT	Lock for report queues	22
KV_LOCK_EMAIL_TASK	Lock for email-related tasks	23
KV_LOCK_HOUSEKEEPER	Lock for housekeeping tasks	24
KV_LOCK_DAV	WebDAV locks	25
KV_SIEVE_ID	Sieve auto-responder tracking ids	26

This structured approach ensures data integrity and prevents key collisions across different types of operations within the in-memory store.

Data Persistence

It is generally not necessary to configure the in-memory store for persistent storage of most key prefixes. Many types of data, such as rate limiter and fail2ban information, are transient and do not require recovery after a server restart.

Configuration

The main in-memory store is defined under the storage.lookup attribute in the configuration file. For example, to use the redis store as the default in-memory store:

[storage]
lookup = "redis"

Although Stalwart requires a default in-memory store, it is possible to define multiple in-memory stores to be used from expressions](/docs/configuration/expressions/overview) and Sieve scripts.

Maintenance

When using a data store as an in-memory store, it is necessary to periodically run an automated task that removes expired keys from the database. The schedule for these tasks is configured using a simplified cron-like syntax. The frequency of these tasks is determined by the store.<id>.purge.frequency attribute of the configuration file, where <id> is the ID of the store you wish to configure.

For example, to run the job every day at 3am local time on the foundationdb store, you would add the following to your configuration file:

[store."foundationdb".purge]
frequency = "0 3 *"