Skip to content

Hosting: backup and restore#

This documentation applies to Piler enterprise edition 2.3.0+

Revision #5

Publication date: 2026-NOV-02

The deployment#

Back up every node as described in large deployments: backup and recovery: the databases, the stored emails (or the S3 buckets), the Manticore indexes, the encryption key, the configuration. The database first, then the emails.

With S3 storage, the node backups don't include the emails: see S3 object storage for versioning, lifecycle rules and replication. A bucket holds one tenant's emails, so these are set up for every new tenant.

With piler-backup.sh#

With the Docker example, piler-backup.sh does this on each node, every day, into a restic repository: the databases of all the tenants, the search index, the stored emails, and what the containers mount from the node's directory (the settings, piler.key, the license). The emails in S3 are not in it.

Set the repository in the node's .env, the example has no setup for it:

BACKUP_REPOSITORY=s3:https://host/bucket/piler     # or sftp:, rest:, a directory
BACKUP_PASSWORD=                                   # openssl rand -hex 32
AWS_ACCESS_KEY_ID=                                 # for an S3 repository
AWS_SECRET_ACCESS_KEY=

Keep BACKUP_PASSWORD off the nodes: without it there's no restore. All the nodes can share a repository, with the same password: each node is a host in it. Then set BACKUP_MAINTENANCE=no on every node but one: that one frees the space and checks the repository on Sundays, which needs the repository alone. Then, as root, in the node's directory, on every node, the master too, each at another time:

piler-backup.sh install-cron 03:15

The retention, the commands and the restore of a whole node to a new host are in Install on a single host. On a node the restore only runs in an installation without piler's volumes.

Admin > Health shows the backups in the node's own database (the backup_status table, migration 030): the master shows the master's, not those of the workers. The PilerBackup* alerts of the monitoring cover every node.

A single tenant#

A tenant is a database, a set of Manticore tables and a bucket (or a directory in the store), on each node: it can be backed up and restored on its own, without touching the other tenants.

Back up#

On each node (the single node, or every worker; the master holds the tenant's settings, users and audit log):

T=customer1

# The database, consistent while piler is running
docker compose exec piler sh -c \
   "mariadb-dump --defaults-file=/etc/piler/.my.cnf --single-transaction $T" > $T-db.sql

And the tenant's Manticore table, without stopping it: FREEZE keeps its files consistent while you copy them (Piler keeps archiving meanwhile), UNFREEZE releases it. In the Manticore container:

mysql -h0 -P9306 -e "FREEZE customer1"
cp -a /var/lib/manticore/customer1 /backup/manticore-customer1
mysql -h0 -P9306 -e "UNFREEZE customer1"

Restore#

Restore the database, the Manticore table and the emails to the same point in time, on every node of the tenant:

T=customer1

# The database
docker compose exec -T piler sh -c \
   "mariadb --defaults-file=/etc/piler/.my.cnf $T" < $T-db.sql

# The Manticore table (in the Manticore container): the backup files are
# copied into Manticore's data directory under the table name
mysql -h0 -P9306 -e "DROP TABLE customer1"
mysql -h0 -P9306 -e "IMPORT TABLE customer1 FROM '/backup/manticore-customer1/customer1'"

The emails: restore the bucket's objects (or the directories /var/piler/store/<server id>/customer1 and /var/piler/astore/<server id>/customer1) to the same point.

Instead of restoring the Manticore table, reindex rebuilds it from the database and the stored emails; slower for a large tenant, but it needs no index backup:

docker compose exec piler reindex -tenantid customer1 -start <min id> -stop <max id> -progress

Restore from a piler-backup.sh backup#

A backup holds the whole node, and piler-backup.sh restore needs an empty installation. For one tenant, extract the backup and restore the tenant's parts. The extract needs room for all of the node's backup, and has to be outside the node's directory. The database dump in it isn't encrypted: remove the directory afterwards.

T=customer1
D=/restore/customer1

piler-backup.sh extract <id> $D
. ./.env

# The database, from the dump of the node's databases. As root: the piler
# user is refused at the first CREATE DATABASE of the dump.
docker compose exec -T -e MYSQL_PWD="$MYSQL_ROOT_PASSWORD" mysql \
   mariadb -uroot --one-database $T < $D/staging/database/mariadb-all-databases.sql

# The Manticore table, into the Manticore container
docker compose exec manticore mkdir -p /backup
docker compose cp $D/var/lib/manticore/piler-backup/backup-*/data/$T manticore:/backup/manticore-$T
docker compose exec manticore chown -R manticore:manticore /backup

Then run the DROP TABLE and IMPORT TABLE of Restore in the Manticore container. Copy the emails from $D/var/piler/store/<server id>/$T and $D/var/piler/astore/<server id>/$T back to the same directories, or restore the bucket.

Erasures after the backup#

Restoring an older state brings back what was removed since: emails a user or a data officer removed, a GDPR erasure, a retention purge. Keep a record of the removals and repeat them after a restore (the audit log has them, see GDPR erasure), and set your backup retention with your compliance obligations in mind.