FoundationDB

In a large Stalwart cluster, every node reads and writes the same data store, so the data store must grow with the cluster and must stay available when hardware fails. FoundationDB is built for that situation. It is a distributed, transactional key-value database that replicates its data across machines and tolerates the loss of some of them, and it scales horizontally by adding machines. Besides the data store, it can also hold the blob store and the search store.

Requirements

A FoundationDB data store has three requirements:

• The official binaries and Docker images do not include FoundationDB support. A separate FoundationDB build of Stalwart is published for Linux, and the installation script installs it with its --fdb option. A Docker image can be built from the Dockerfile.fdb in the Stalwart repository.
• The FoundationDB client library (libfdb_c) must be installed on every Stalwart node, in a version compatible with the FoundationDB servers.
• Every node needs the cluster file of the FoundationDB cluster, which lists its coordination servers. FoundationDB’s administration guide describes how to distribute it.

FoundationDB also places limits on what a single operation may do. Each transaction may last at most five seconds and carry at most 10 MB of data, and each value is limited to 100 KB. Stalwart splits large values and blobs into chunks and writes large blobs over several transactions, so no configuration is needed for large messages.

Configuration

The backend is selected with the FoundationDb variant of the DataStore object (found in the WebUI under Settings › Storage › Data Store). The same fields apply to the FoundationDb variants of the BlobStore and SearchStore objects. Every field is optional, and a field left unset keeps the FoundationDB client default.

The first setting tells the client how to find the cluster. clusterFile is the path of the cluster file. When it is unset, the client looks in its default location, which on Linux is /etc/foundationdb/fdb.cluster.

The next settings bound how long the client keeps trying. transactionTimeout is the time a transaction may take, in milliseconds. A transaction that fails with a retryable error is retried by the client, and transactionRetryLimit and transactionRetryDelay limit how often it is retried and the maximum delay between attempts.

The last two settings matter only in deployments that span several data centres. machineId and datacenterId are locality options passed to the FoundationDB client. They tell FoundationDB on which machine and in which data centre the client runs, and FoundationDB uses this information to route requests in multi-data-centre deployments.

The following data store uses a cluster file in a non-default location:

{
  "@type": "FoundationDb",
  "clusterFile": "/etc/stalwart/fdb.cluster"
}