Overview

A mail server keeps several kinds of data, and they behave very differently. Account records and mailbox state are small, and the server reads them constantly. Message contents are large, and once written they rarely change. The search index can be rebuilt from the other two, while rate-limit counters and locks matter only for a few seconds or minutes. A single storage system that suits one of these patterns is a poor fit for the others.

Stalwart therefore keeps each kind of data in its own store, and each store is configured independently. A small deployment can keep everything in one embedded database on a single server, with no other service to run. A larger one can place each kind of data on the service best suited to it, for example the messages on object storage and the counters on Redis.

Main stores

The data store holds every structured record of the server. This covers accounts and domains, mailboxes and message metadata, calendars and contacts, the change history from which clients synchronise, and the server’s configuration objects. It is the only store that every other part of the server depends on. It is configured through the DataStore object (found in the WebUI under Settings › Storage › Data Store).

The blob store holds large binary objects, or blobs: the raw messages with their attachments, Sieve scripts, the contents of file storage and similar data. Every blob is written to the blob store. With the Default variant, however, the blob store keeps its blobs inside the data store, so no separate service is needed until the volume of data calls for one. It is configured through the BlobStore object (found in the WebUI under Settings › Storage › Blob Store).

The search store holds the full-text index of messages, calendar events, contacts and files, which answers text searches. The Default variant keeps the index in the data store, and ElasticSearch or Meilisearch can hold it instead. Documents are not indexed at the moment they are stored but in the background, as described in Search indexing. It is configured through the SearchStore object (found in the WebUI under Settings › Storage › Search Store).

The in-memory store holds short-lived data: rate-limit counters, greylisting entries, distributed locks, and the tokens used during OAuth and ACME exchanges. The Default variant keeps this data in the memory of the server process. That suits a single node, although the data is lost on restart. A cluster, whose nodes must see the same counters and locks, requires a Redis-compatible in-memory store that all nodes share. It is configured through the InMemoryStore object (found in the WebUI under Settings › Storage › In-Memory Store).

Each of these objects is a multi-variant object. Its @type selects the backend, and the fields of that variant configure it. Storage backends compares the backends available for each store.

Two rules apply to every store. First, changing a store’s backend only changes where the server looks for its data: it does not move the existing data, so moving it is a separate task, described in Database migration. Second, changes to any store take effect only after a restart.

Other stores

A few other settings also decide where data is kept. The cache swap tier persists snapshots of account caches. With it, an account does not have to be rebuilt from the data store after a restart or when it is opened on another node. It has no effect until it is enabled on the Cache object.

When telemetry history is enabled, the TracingStore and MetricsStore objects select where the delivery history and the metrics history are kept.

Lookup lists are lists of keys and values, such as domain blocklists, that expressions and Sieve scripts query. They are not kept in any of the stores above: the server holds them in memory and loads them from the configuration or from remote URLs.

Data retention and disk space

Deleted and expired data is not removed at once. Purge processes remove it on a schedule, and each backend returns the freed disk space in its own way. Data retention and reclaiming disk space describes how long the data is kept, when the purges run and when each backend returns the space. The retention periods and schedules are set on the DataRetention object.

Recovering deleted items

Deleted messages, files, calendar events, contacts and Sieve scripts can be restored from the deleted-item archive, as described in Recovering deleted items.