Skip to content

Upgrading#

This documentation applies to Piler enterprise edition 2.3.0

Revision #8

Publication date: 2026-NOV-02

Upgrade steps#

Run the migrations#

Run mysql migrations on each piler related mysql database:

for i in /usr/share/piler/migrations/sql/*.sql; do
   mariadb --defaults-file=/etc/piler/.my.cnf piler < $i
done

Run manticore migrations for each tenant: In case of a multi-tenant deployment, be sure to use the tenant id

for i in /usr/share/piler/migrations/manticore/*.sql; do
   sed 's/TENANTID/piler/g' "$i" | mariadb -h0 -P9306
done

The migrations are safe to run again: run all of them at every upgrade, those of earlier versions too.

For a multi-tenant deployment, run the mysql migrations for all piler databases, also run the manticore migrations for all tenant ids as well, and replace "piler" in the sed command with the tenant id, eg.

sed 's/TENANTID/example001/g' ....

Migration 030 (2.3.0) creates the backup_status table the admin health page reads the backups from. Only the table in the piler database is used, the copies in the tenant databases stay empty. Until it has run, the page shows no backups, nothing else depends on it.

The Docker images don't run it. piler-upgrade.sh of the Docker examples runs the migrations a new version adds (piler-examples of 2026-11-02 or later, git pull it first). Otherwise run the migrations in the piler container (docker compose exec piler sh), or this one alone, on every node:

docker compose exec -T piler sh -c \
   "mariadb --defaults-file=/etc/piler/.my.cnf piler < /usr/share/piler/migrations/sql/030_backup_status.sql"

A native 2.x install is upgraded with the packages and these migrations. contrib/installer/upgrade-from-1.8.4.sh (upgrade.sh before 2.3.1) is the upgrade of a 1.8.4 install, run once, see Upgrading from 1.x.

Manticore 25.0.0 or newer (2.3.2)#

From 2.3.2 piler needs Manticore 25.0.0 or newer, on every node that runs piler-ui; 29.9.0, the version piler is tested with, is recommended. With 25.0.0 or newer nothing has to be done now, an upgrade to 29.9.0 can wait for a convenient time. The queries of piler-ui pass their values to Manticore separately from the query, which the older versions don't support: with one, piler-ui doesn't start, and logs Manticore X is too old, piler needs 25.0.0 or newer. The scanner and the Slack collector log the same error, then save no semantic search data and purge no Slack messages until Manticore is upgraded.

Check the version before the upgrade:

# native install
mariadb -h0 -P9306 -e "SHOW STATUS LIKE 'version'"

# Docker
docker compose exec manticore mysql -h0 -P9306 -e "SHOW STATUS LIKE 'version'"

If it's older than 25.0.0, back up the index first (see Backup), check the Manticore release notes of the versions in between, then upgrade Manticore, preferably to 29.9.0 or newer:

  • Native install: apt-get update && apt-get install manticore (Debian, Ubuntu) or yum update manticore (RHEL, Oracle Linux), from the Manticore repository the installer set up.
  • Docker: set the image of the manticore service in docker-compose.yaml to manticoresearch/manticore:29.9.0, the version of the current Docker examples (at least 25.0.0), and run docker compose up -d manticore. The examples before 25 October 2026 had older versions, e.g. 13.11.0 or 15.1.0.

Upgrade Manticore first, then piler.

Slack tables: deleted_at (2.3.2)#

A Slack table created by the Manticore migrations, e.g. at an upgrade from 1.x, had no deleted_at column, and the Slack purge failed for every message of it. After the migrations run:

/usr/libexec/piler/add-missing-columns.sh 0 --all

It's the script of the participants column, renamed from add-participants-column.sh: it adds both columns where they are missing, and skips a tenant without a Slack table, so it's safe to run again. On a multinode master run it with --no-index, which leaves the distributed index alone and adds the column to the Slack tables only:

/usr/libexec/piler/add-missing-columns.sh --no-index 0 --all

With Docker there's nothing to do: the piler-ee container runs the script at every start, with --no-index on a master.

Date formats (2.3.1)#

The date format in the settings of a user is a format of dayjs tokens, e.g. YYYY/MM/DD or DD.MM.YYYY HH:mm. Users upgraded from 1.8.x may still have a PHP format there (d.m.Y, Y-m-d H:i), and their dates were garbled. Migration 033 resets every format without a year (YY or YYYY), a month (M) and a day (D) to YYYY/MM/DD, in the piler database and in the database of every tenant. The users can set their format again in their settings.

A format with other text than these tokens and separators, e.g. YYYY年MM月DD日, is reset too and can't be saved.

Quotas (2.3.0)#

Multi-tenant deployments only.

  • The user count is the active users. user_count, the number the user quota applies to, is now the addresses of the tenant's domains that received email in the last 90 days; before, it was every address that ever received email. It drops for tenants with former employees: check the user quotas on the quota dashboard after the upgrade. Every address is still listed as archived_user_count. See billing.
  • Imported mail: from 2.3.0 pilerimport counts the date of each email. The addresses of mail imported before the upgrade count with the time of the import, so they stay active up to 90 days after the import.
  • Multi-node: upgrade the workers before the master. The master asks the workers for their active and archived addresses. An older worker sends only every address it has: the master counts all of them as active, and logs a warning, until the worker is upgraded.
  • The usage is at most 15 minutes old. The quota dashboard and /api/v1/admin/quota/all show the usage the master collects every QUOTA_SNAPSHOT_INTERVAL (default 15 minutes), with the time it was collected; a new tenant is "pending" until the next collection. Billing scripts reading quota/all get the same numbers.
  • Several masters: set LEADER_ELECTION_ENABLED=true on all of them, with the same Redis. Otherwise each collects the usage of every tenant and sends the quota warning emails. See several master nodes.
  • The quota warning emails are sent by the master only (the leader with several masters), not by the workers.
  • Billing API tokens: run the SQL migrations on the master's piler database (028_api_token.sql creates the api_token table). Billing scripts that log in with the admin's password can switch to an API token, see billing.

Metrics through a reverse proxy (2.3.0)#

/metrics and /healthz answer only the addresses in ALLOWED_CIDRS. A request through a reverse proxy now counts only if the proxy is in TRUSTED_PROXIES, and then the client is the address in its PROXY_HEADER (default X-Forwarded-For); a forwarded request from any other address is refused. A scraper connecting to piler-ui directly isn't affected. If yours goes through a proxy, add the proxy to TRUSTED_PROXIES, or better, scrape piler-ui directly. See Hosting: monitoring.

Docker: the error directory volume (2.3.0)#

The Docker examples mount the error directory (/var/piler/error, the emails piler couldn't archive and the files s3uploader couldn't upload) as the piler_error volume, in the piler and the s3uploader containers. Before, it was in the containers, and lost when they were recreated.

If you take the new docker-compose.yaml, before the upgrade check that the directory is empty, move what's there back for processing, then create the volume, otherwise the containers don't start:

docker compose exec piler ls /var/piler/error
docker compose exec s3uploader ls /var/piler/error
docker volume create piler_error

On a multi-node deployment do it on every worker.

Add the participants column (2.3.0)#

Piler 2.3.0 adds the participants attribute to the index. piler, piler-ui, pilerimport and reindex check it at startup, and refuse to start until the column is there. The package upgrade stops the piler services, so back up the Manticore data, then run the script before starting them again:

/usr/libexec/piler/add-missing-columns.sh 0 --all

Before 2.3.2 the script was add-participants-column.sh.

The first argument is the Manticore host (0 is the local host). --all means the piler index and the index of every tenant in the tenant_settings table, so the same command works for single-tenant and multi-tenant deployments.

It's enough to run it once, but it's safe to run it again: it adds the column only to the indexes that don't have it yet, and leaves the others alone.

An index can't be searched while the column is added, but it's quick: on an index of 580,000 messages (1.1 GB, 4 disk chunks) it took less than a second. Only the attribute files are changed, not the full-text index.

With Docker there's nothing to do: the piler-ee container runs the script at every start.

In a multinode setup upgrade the workers first, and run the script on every worker; on the master only with --no-index (2.3.2): its index tables are distributed. Start piler-ui on the master only when all the workers have the column, otherwise it refuses to start.

If piler-ui on the master still refuses to start, and names a tenant the workers have the column of, that tenant has local tables on the master. Check with SHOW TABLES on the Manticore of the master: <tenant>, <tenant>_tag1, <tenant>_note1 and <tenant>_embeddings must be of type distributed (_audit and _cat1 are local). The searches of a tenant with local tables there don't reach the workers. Check that the local ones are empty (SELECT COUNT(*) FROM <table>), drop them, and create the distributed tables:

mariadb -h <manticore host> -P9306 -e "DROP TABLE <tenant>"   # and each local _tag1, _note1, _embeddings
/usr/libexec/piler/recreate-distributed-tables.sh <manticore host> <tenant>

Adding the column to the local tables lets piler-ui start, but leaves the searches of the tenant broken.

Optionally fill the attribute of the messages archived before the upgrade:

reindex -tenantid piler -start 1 -stop <max metadata id> -participants -progress

It reads the addresses from the database, not the messages, can run while piler is archiving and users are searching, and can be interrupted and run again. Without it the messages archived before the upgrade are searched the same way as before.

S3 storage settings (2.3.0)#

Only for S3 storage (RETRIEVER_METHOD=s3).

s3uploader reads /var/piler/.env. Before 2.3.0 the piler-s3 service read its settings from /var/piler/.env-s3, a second copy of the S3 settings that are in /var/piler/.env anyway (piler, piler-ui, purge, reindex and pilerexport read them from there). From 2.3.0 it reads /var/piler/.env like the others, and .env-s3 is not read any more. Before starting piler-s3 after the upgrade:

  1. Move the settings only s3uploader uses from .env-s3 to .env, if you have any: S3_WORKERS, S3_SCAN_INTERVAL, S3_SKIP_EXISTING_OBJECTS, S3_UPLOAD_MAX_RETRIES, S3_UPLOAD_RETRY_DELAY, S3_UPLOAD_TIMEOUT, S3_FILE_STABILITY_DELAY, S3_VERIFY_AFTER_UPLOAD, S3_PART_SIZE_MB, S3_DISABLE_MULTIPART, PILER_S3_DIR. Without them the defaults apply.
  2. s3uploader now uses the credentials in .env. They need write access to the buckets (purge needs to delete with them already). If .env-s3 had other credentials for uploading, use those in .env.
  3. Don't move S3_BUCKET_PREFIX or S3_PREFIX, see below.
  4. Remove /var/piler/.env-s3. s3uploader logs a warning while it exists.

The bucket. piler-ui, purge, reindex and pilerexport read the emails of a tenant from the bucket S3_BUCKET_PREFIX + tenant id. Before 2.3.0 s3uploader ignored S3_BUCKET_PREFIX and always uploaded to the bucket named after the tenant id (piler on a single-tenant install). With S3_PREFIX set it also put the objects under that key prefix, where nothing reads them. From 2.3.0 s3uploader uploads to S3_BUCKET_PREFIX + tenant id too, and refuses to start with S3_PREFIX set.

With S3_BUCKET_PREFIX empty in /var/piler/.env and no S3_PREFIX in either file, the default, there's nothing more to do. Otherwise:

Setting Before 2.3.0 What to do
S3_BUCKET_PREFIX set in .env The uploaded emails could not be read Either remove S3_BUCKET_PREFIX, or copy the objects to the prefixed bucket, see below
S3_BUCKET_PREFIX set in .env-s3 only Ignored, everything used the tenant bucket Nothing, it's not read any more
S3_PREFIX set in .env-s3 (or .env) The uploaded emails could not be read Copy the objects out of the key prefix, see below. Make sure .env has no S3_PREFIX

Removing S3_BUCKET_PREFIX is the simplest: the objects are already in the tenant bucket, nothing has to be copied. Keep the prefix only if the buckets must have the prefixed names.

To copy the objects, first do the steps above and start piler-s3, so the new emails already go to the right place. The key of an object is the same, without the S3_PREFIX part: <2 characters>/<file name>. Then copy, for each tenant id (piler on a single-tenant install), e.g. with the AWS CLI:

# S3_BUCKET_PREFIX was ignored: from the tenant bucket to the prefixed one
aws s3 mb s3://<S3_BUCKET_PREFIX><tenant id>
aws s3 sync s3://<tenant id>/ s3://<S3_BUCKET_PREFIX><tenant id>/

# S3_PREFIX was set: from under the key prefix to the top of the bucket
aws s3 sync s3://<tenant id>/<S3_PREFIX>/ s3://<S3_BUCKET_PREFIX><tenant id>/

Add --endpoint-url https://<S3_ENDPOINT> for a store other than AWS. With the MinIO client it's mc mirror with the same source and target. sync can be run again: it copies only what's missing. aws s3 mb fails if the bucket exists already, that's fine.

Open a few emails archived before the upgrade in piler-ui to check them. Delete the old objects only after that, and after your backups have the new bucket.

The billing server URL (2.3.0)#

From 2.3.0 there is one billing server, and the default BILLING_SERVER_URL is https://billing.mailpiler.com. If you set BILLING_SERVER_URL by hand to https://billing1.mailpiler.com,https://billing2.mailpiler.com (the old default), replace it with https://billing.mailpiler.com, or remove the setting. The old names still reach the same server, so nothing breaks meanwhile, but a failed report is then sent to the same server twice.

E-invoices in the search (2.3.0)#

From 2.3.0 Piler marks the emails with an e-invoice attachment when it archives them, and attachment:einvoice finds them. To find the e-invoices archived before, rebuild their index entries with reindex, e.g. for the emails of the last 10 years:

reindex -tenantid piler -start <first metadata id> -stop <max metadata id> -progress

reindex reads the stored emails. Without attachment extraction (EXTRACT_ATTACHMENTS, on when TIKA_URL is set) it also reads the attachments that may be e-invoices (XML and PDF), one at a time until it finds one, from the store or the S3 bucket: plan for the extra reads on a large archive. Searches keep working meanwhile; an entry is replaced when it's rebuilt.

Audit log details (2.3.0)#

The string values in the details of the audit entries are quoted from 2.3.0 (username="anna", was username=anna), and the legal hold entries have keys: case_number="…", case_name="…" (was CASE-1 - Name) and hold_id="…", reason="…" (was Hold ID: … - Reason: …). If a SIEM rule or a report matches the old text of these details, update it.

Refresh sessions (2.3.0)#

Before 2.3.0 a token login (POST /auth/login) stored the user's TOTP secret and backup codes, IMAP password, access token and CSRF token with the refresh session in the refresh_session table. Migration 029 removes them from the stored sessions, and deletes the expired ones; piler-ui does the same daily in every tenant database. The sessions stay valid, nobody is logged out.

The cleanup doesn't reach the copies of the table in your database backups and binary logs. If these may have been exposed, have the users with a token login reset their TOTP (2FA), and those logging in with IMAP authentication change their mailbox password.

JWT secret (2.3.0)#

Every deployment, a single node too, needs a JWT_SECRET of its own: before 2.3.0 piler-ui needs it to refuse access tokens it didn't issue. If you can't upgrade right away, set it now.

piler-ui signs its access tokens with JWT_SECRET: the tokens of the REST API and the Outlook add-in, and those a multi-node master sends to its workers. Set it to a random value of at least 32 characters, e.g.

openssl rand -hex 32

in /var/piler/.env (the Docker examples: env on a single node, .env on each node of the multi-node example). Use the same value on every node of a multi-node deployment. Replace a value copied from an example or from this documentation, a changeme one too. Until 1 November 2026 this documentation printed a fixed JWT_SECRET for native multi-node installs: from 2.3.1 piler-ui warns about that value at startup.

Without it, piler-ui uses a random secret until it restarts, and logs a warning. The web UI works, but the REST API tokens and the Outlook add-in need a new login after each restart, and a multi-node master can't query its workers.

Restart piler-ui after setting it. REST API clients and the Outlook add-in log in again. On a multi-node deployment a user's searches may fail for up to 15 minutes, until their token is renewed: logging out and in again fixes it at once.

Retention settings (2.3.0)#

The options table. Before 2.3.0 it had copies of retention settings, in the piler database and in the database of each tenant. Migration 032 removes them:

  • enable_purge and retention_days: nothing read them. The settings are the columns of piler.tenant_settings, as before.
  • enable_tenant_retention_policies and min_retention_days: they're ENABLE_TENANT_RETENTION_POLICIES and MIN_RETENTION_DAYS in the .env of piler-ui now, see Config options. If you had changed one, the migration prints it, e.g. Move to the .env of piler-ui: MIN_RETENTION_DAYS=90: add the line to the .env of piler-ui (on a multi-node deployment, of the master).
  • retention_grace_period_days, retention_require_confirmation and retention_recovery_days: gone.

Retention rules. A rule is active when saved, and the migration activates the pending ones (waiting for their grace period or a confirmation). This changes nothing for the archiving: the piler daemon applied the pending rules already. The confirm API (POST /api/v1/admin/retention/rule/:id/confirm) is gone: remove its call from your scripts.

Purge. purge reads enable_purge of the tenant from piler.tenant_settings of the node it runs on, and stops with an error for a tenant without a row there, or when it can't read it: before 2.3.0 it purged then. On a multi-node deployment the purge runs on the workers, so the tenant's row on each worker decides, not the master's. Check it on every worker:

docker compose exec mysql mariadb -u piler -p piler \
   -e "SELECT tenant_id, retention_days, enable_purge FROM tenant_settings"

and set a tenant's settings on each node, see Change a tenant's settings.

The default retention. A new installation keeps the emails for 10 years (3653 days) by default, it was 20 years (7300 days). An existing installation keeps its retention: on a single node it's retention_days of the piler row above, counted from the date of each email when it's archived. To change it, UPDATE tenant_settings SET retention_days=3653 WHERE tenant_id='piler'; the emails archived before keep theirs.