Overview

Telemetry is how an administrator finds out what the server is doing: why a message was rejected, whether the outbound queue is growing, or when a database started returning errors. Stalwart reports its activity as events. An event is a single thing that happened, such as an accepted connection, a delivered message, a failed authentication or a rebuilt cache.

Every event has a name, a severity level and a set of key-value pairs that describe it. From these events the server also derives metrics: counters and other measurements that summarise activity over time, such as the number of messages delivered.

The telemetry facilities decide where events and metrics go, and each serves a different purpose:

• Tracers write events to a log file, the console, the systemd journal or an OpenTelemetry collector. Each tracer has its own level and event filter. The events and their levels are described in Events.
• Metrics are exported to Prometheus or an OpenTelemetry collector, for an existing monitoring system.
• Webhooks post selected events to an HTTP endpoint, so that other systems can react to them.
• Alerts evaluate conditions on metrics and send an email or emit an event when they hold (Enterprise).
• Live telemetry streams events and metrics to the WebUI or another HTTP client in real time (Enterprise).
• History stores message delivery traces and metrics in a database, for the WebUI’s delivery history and dashboard (Enterprise).

Logs, stored traces and stored metrics can be read back from the WebUI, the CLI and the API. Telemetry management describes the objects through which this is done.

Recommended setup

A production server typically needs four things.

First, one tracer at level info. It writes to a log file, or to the console when the container runtime or systemd collects the output. A log file has an extra advantage: the WebUI’s log viewer reads the files of a Log tracer, so a Log tracer is needed to view logs there.

Some details that administrators often look for are logged at debug, below that level: per-attempt delivery details, authentication failures and spam filter verdicts, as listed in Events. Raising those individual events to info is preferable to running a whole tracer at debug, because the log stays readable.

Second, metrics exported to the monitoring system in use, through the Prometheus endpoint or an OpenTelemetry collector.

Third, monitoring of a few signals that indicate trouble:

• The size of the outbound queue (queue.count).
• Store errors, such as store.postgresql-error or store.s3-error.
• The memory used by the server (server.memory).
• The telemetry.events-dropped event, which reports that tracers could not keep up and that the logs are incomplete.

With the Enterprise edition, the same conditions can be configured as alerts, without an external monitoring system.

Fourth, for the WebUI dashboard and delivery history, the MetricsStore and TracingStore (Enterprise). A new installation keeps both in the data store, where they take up space. Plan for that space, or set them to Disabled.

Measuring performance describes which metrics to watch when investigating performance.