Remote lists

A remote lookup list is downloaded from a URL, and held in memory. It suits lists maintained outside the server, such as public lists of disposable email domains or phishing URLs, or a list exported by another internal system. The list is downloaded again at a fixed interval, so changes at the source reach the server without any configuration change.

Downloads and refresh

A list is downloaded in the background the first time it is queried, so queries made before the first download completes find the list empty. After a successful download, the list is kept for 12 hours, as set by refreshInterval. The next query after that starts a new download. The previous content stays in use until the download completes. Each successful download is logged as a store.http-store-fetch event with the number of entries.

A failed download does not empty the list, so a source that is briefly unavailable does not disable the checks that use it. The previous content is kept, the error is logged as a store.http-store-error event, and the download is attempted again after 1 hour, as set by retry.

A remote list is queried with key_exists, or with key_get when it is a CSV file with a value column. Keys are compared without regard to case.

Configuration

A remote list is defined by an HttpLookup object, found in the WebUI under Settings › Lookups › HTTP Lists. Two fields identify the list. Its namespace is the name under which the list is queried, and must be unique among all lookup lists. Its url is the address of the file. enable turns the list on or off, and isGzipped decompresses a file served compressed with gzip.

The remaining settings describe the HTTP request. A list served by a protected endpoint needs credentials, given in httpAuth: Basic with username and secret, or Bearer with bearerToken. A public list uses Unauthenticated.

httpHeaders adds headers to the request, for example an API key header. When no User-Agent header is given, a browser user agent is sent, because some public list providers reject other clients. timeout limits the duration of each download. allowInvalidCerts accepts an invalid TLS certificate.

Formats

Lists are published in two common shapes: a plain file with one entry per line, and a CSV file with several columns. The format field selects how the file is read:

• List: one key per line. Empty lines are ignored, and surrounding whitespace is removed.
• Csv: one entry per line, split into columns at separator. indexKey is the zero-based column that holds the key, and indexValue is the column that holds the value. Without indexValue, the list contains keys only. skipFirst skips a header line.

The following object downloads a public list of disposable email domains once a day:

{
  "namespace": "disposable-domains",
  "url": "https://disposable.github.io/disposable-email-domains/domains.txt",
  "format": {"@type": "List"},
  "httpAuth": {"@type": "Unauthenticated"},
  "refreshInterval": 86400000
}

Limits

Three limits protect the server from oversized files. Each one decides what happens to a list that outgrows it, so a large public list may need them raised:

Field	Default	Limits	Beyond the limit
maxSize	100 MiB	The size of the download and the length of each line.	A larger file is rejected with the error Resource is too large, and the previous content of the list is kept. A longer line is skipped.
maxEntries	100000	The number of entries.	Reading stops, and the entries read up to that point are used.
maxEntrySize	512 bytes	The length of a key or a value.	The line, key or value is skipped, and never stored truncated.

Compression changes what maxSize measures. For a file compressed with gzip, maxSize applies to the compressed download, so the line length cap is the only size limit on its decompressed content.