Skip to content

Large deployments: architecture and high availability#

This documentation applies to Piler enterprise edition 2.2.4+

Revision #1

Publication date: 2026-OCT-18

The recommended architecture of a site in a large, multi-node Piler deployment: tens of thousands of users or more, usually many tenants. For a second site, see disaster recovery.

Layout of a site#

          Your mail server (a copy of every email, by SMTP)
                       │
                ┌──────▼──────┐
                │  MX relay   │  optional: spreads the emails among the
                └──────┬──────┘  workers, queues while one is down
        ┌──────────────┼──────────────┬──────────────┐
  ┌─────▼────┐   ┌─────▼────┐   ┌─────▼────┐   ┌─────▼────┐
  │ worker 1 │   │ worker 2 │   │ worker 3 │   │ worker 4 │   piler-smtp, piler daemon,
  └──────────┘   └──────────┘   └──────────┘   └──────────┘   MySQL, Manticore, stored emails
        ▲              ▲              ▲              ▲
        └──────────────┴──────┬───────┴──────────────┘
                     ┌────────┴────────┐
                     │  load balancer  │  ◄── users, REST API clients
              ┌──────┴─────┐    ┌──────┴─────┐
              │    UI 1    │    │    UI 2    │   piler-ui; MySQL, Manticore (audit), Redis
              └────────────┘    └────────────┘
                 smarthost for restores ──► your mail server
  • Worker nodes receive the emails (piler-smtp) and archive them. Each worker has its own MySQL database, Manticore search index and email store, and keeps the emails it archived. The number of workers is set by your mail volume and retention, see sizing.
  • UI nodes run the web UI and the REST API. They query the workers for search results and messages, so users see one archive. Several UI nodes run behind a load balancer; background jobs run on one of them at a time, chosen automatically (leader election).
  • MX relay (optional, needs no Piler license): spreads the emails from your mail server among the workers. Your mail server can also send to the workers directly, e.g. with several MX records.
  • Smarthost: delivers restored emails back to the users' mailboxes through your mail server.

All Piler components on the same server (piler-smtp, the archiving daemon, the web UI) share one license.

The number of UI nodes depends on how many users search at the same time. Two give redundancy; more can be added at any time as the load grows.

The encryption key#

Piler encrypts every stored email with the key in piler.key. Use the same piler.key on every node of the deployment, the workers and the UI nodes of both sites. It's the best practice for a multi-node deployment, and a requirement when the emails are stored in shared storage such as an S3 bucket: any node, and a node restored or moved to another site, can then read every stored email.

Keep a copy of the key outside the deployment, separately and securely: without it the archive can't be read, by anyone.

High availability#

Component Redundancy How
Web UI and REST API Yes Several UI nodes behind a load balancer. Their MySQL database should be highly available too, behind a single endpoint (e.g. a database proxy or a virtual IP). Redis holds sessions and caches only: if it's lost, users log in again.
Receiving email Yes The emails are spread among the workers; if a worker is down, the others keep archiving, and the MX relay or your mail server queues what can't be delivered.
MX relay (if used) Yes Two relays, behind DNS or your mail server's routing.
Archived emails of a worker At the infrastructure level Each worker keeps the emails it archived. While a worker is down, its emails can't be searched or viewed at the production site until it's back. See below.

Protecting the workers#

Each worker holds three kinds of data: its MySQL database (the metadata and recipients of its emails), its Manticore search index, and the stored emails. All three are needed to find and open an email. Piler doesn't replicate them to another worker of the same site, so the workers are protected at the infrastructure level, in layers:

  1. The data survives a host failure. Keep the database, the search index and the stored emails on replicated network storage, e.g. block volumes on Ceph (OpenStack Cinder with Ceph RBD). The stored emails may be in S3-compatible object storage instead. Object storage alone is not enough: the database and the search index stay on the worker.
  2. The worker comes back on another host. Your virtualization platform restarts a failed worker VM on another host with the same volumes (e.g. OpenStack instance high availability, or an evacuation). The worker returns with all its data, typically within minutes. Meanwhile the other workers keep archiving new email; only the emails of the failed worker are not searchable.
  3. Backups, if the data itself is damaged, see backup and recovery.
  4. The DR site, which holds a complete copy, if the primary site is down as a whole.

Avoid the local disks of the hypervisor hosts for the workers: if a host fails, its local disks go with it, the worker VM can't be restarted elsewhere, and its emails are unavailable until the host is repaired, or must be restored from the backups.

Storage#

  • Stored emails: replicated block storage volumes on each worker, or S3-compatible object storage (e.g. Ceph RGW, MinIO). If you plan to use another object store through an S3 API, we recommend testing it in the proof of concept.
  • MySQL and Manticore on each worker: fast (SSD), replicated block storage volumes, not the local disks of the hypervisor hosts (see protecting the workers).
  • Volumes: see sizing.