Skip to content

Large deployments: backup and recovery#

This documentation applies to Piler enterprise edition 2.3.0+

Revision #4

Publication date: 2026-NOV-02

Backups are implemented with the backup system you already run. What to back up:

What Where Notes
MySQL / MariaDB databases every worker and UI node The metadata of every archived email, the users, the tenant settings. Back up consistently: a hot backup (e.g. mysqldump --single-transaction or mariabackup), or binary log shipping for point-in-time recovery.
Stored emails and attachments every worker (/var/piler/store, /var/piler/astore), or the S3 bucket Millions of small, write-once files: incremental file backups, filesystem snapshots, or object storage versioning / replication.
Manticore search index every worker Can be rebuilt from the archive, but at this scale that takes long: back it up as well.
Manticore audit log the UI nodes Every access to the archive. Keep it as long as your compliance rules require.
The encryption key, piler.key the same on every node Without it the archive can't be read. Keep a copy separately and securely, outside the regular backups too.
Configuration and license every node The environment files, the Docker Compose files, the license file.

Stored emails change only in recent directories. The emails are stored as /var/piler/store/<server id>/<tenant>/<xxx>/..., where <xxx> comes from the time the email was archived: a new <xxx> directory is started about every 12 days, and no new email is written to an older one. So after a full backup, only the directories of the last 12 days need to be backed up again; the older ones only change when emails are purged from them. Attachments (/var/piler/astore) are stored by their content, so new files can appear in any of their directories; as every file is written once and never changed, an incremental backup handles them efficiently.

Purged and erased emails stay in the backups until the backups themselves expire: set your backup retention with your compliance rules (e.g. GDPR erasure) in mind.

Order: back up a worker's database first, then its stored emails. Then every email in the database backup has its stored file in the backup; the files of emails archived in between are only extra. The other way round, files older than the database would leave the newest emails without their content. With a DR site, emails archived after the last backup can be recovered from it, as it received them as well.

Restore: restore a worker's database, search index and stored emails to the same point, with the same piler.key, then start its containers. While a site is being restored, a DR site can serve the users.

S3 object storage#

With S3 storage the emails are not on the nodes: a file backup or piler-backup.sh of a node doesn't include them, only those still waiting for the upload (/var/piler/s3). Protect the buckets in the object storage itself. A restic or file backup of the buckets would be a second full copy, with its transfer costs.

Protection Covers Doesn't cover
Versioning An accidental delete, a bad purge, damage through piler's S3 key Losing the provider, the account or the site
Replication to another region or provider Losing the storage system Deletes, unless the target is versioned too
Object lock Tampering, even by an administrator It conflicts with the purge and with GDPR erasure

Use versioning and replication together. For each bucket, the tenant buckets and the Teams and Slack ones:

  1. Enable versioning before the tenant receives email. The purge then only adds a delete marker, and the email can be recovered by removing it.
  2. Add a lifecycle rule that expires the old versions, e.g. after 30 to 90 days (NoncurrentVersionExpiration), and removes the delete markers left without a version. Without it purged and erased emails stay in the bucket forever, and it only grows.
  3. Replicate it to another region or provider, with the same lifecycle rule on the target, so that erasures reach the copy too.
  4. Keep the S3 key of piler limited to the permissions it needs. Without s3:DeleteObjectVersion and the lifecycle permissions, a stolen key can't remove the old versions.

Use object lock only if your rules require WORM storage, and then with a short lock, e.g. on the replica: a locked object can't be purged or erased.

Object stores differ in what they support, and in the names of these settings: check versioning, lifecycle rules and replication with yours before you rely on them, and try restoring a deleted object.

Erasure: an erased email stays in the old versions until the lifecycle rule expires them, and in the replica until its rule does. Set the period with your compliance rules in mind.

Restore: restore the node as usual, and leave the bucket as it is. The emails archived after the database backup stay in the bucket, but the restored database doesn't refer to them: they're not in the archive until they're imported again. An object removed by mistake comes back by removing its delete marker.

If the database backup is older than the lifecycle period, it may refer to objects that are gone: emails purged or erased since the backup. These can't be opened, and they were past their retention anyway. Repeat the erasures made since the backup, see GDPR erasure.

The backup script of the Docker examples#

If you have no backup system for it, the Docker examples have piler-backup.sh: run on every node, it backs up everything in the table above into a restic repository every day, the database first, except the emails in S3: protect those with your object storage's versioning or replication. See hosting: backup and restore.

The database backup is a logical dump, and the recovery point is the last daily run. For a database too large to dump every night, or a shorter recovery point, use a hot backup or binary log shipping.

The admin health page#

Admin > Health shows the backups in the backup_status table of the node's piler database (migration 030). piler-backup.sh writes it; another backup system can insert the same rows after each backup:

INSERT INTO piler.backup_status
  (backup_type, backup_timestamp, backup_size_bytes, status, verification_timestamp, verification_status)
VALUES
  ('database', UNIX_TIMESTAMP(), 5368709120, 'success', NULL, NULL);

backup_type is database, manticore or email_store, status is success or failed, verification_status is pass, fail or NULL (not verified).

On a node with S3 storage email_store says only that the node's backup worked, the size is that of the node's backup: the page says nothing about the buckets.

The page reads the database of its own node: the master shows the backups of the master, not of the workers. piler-ui's /metrics has the backups of every node (piler_backup_last_timestamp_seconds, piler_backup_failed, piler_backup_verification_failed): alert on these, see the monitoring example.