Skip to content

Retention and exclusion rules#

How long piler keeps an email, and which emails it doesn't archive at all. The admin manages both on Compliance > Retention Policy.

How the retention of an email is set#

When piler archives an email:

  1. Exclusion rules. If an enabled exclusion rule matches, the email is not archived, see Exclusion rules.
  2. Retention rules. The first enabled retention rule that matches gives the number of days. The rules are checked in the order they were created, the oldest first.
  3. Default retention. If no rule matches, the default retention of the tenant applies (10 years on a new install).

The retention date of the email is its Date header plus these days. It's stored with the email and doesn't change later: a new or changed rule, or a new default retention, applies to the emails archived from then on, not to those already in the archive.

After its retention date the purge removes the email, see Purge.

Default retention#

The days of the emails no retention rule matches: retention_days of the tenant in piler.tenant_settings.

  • Single tenant: the retention page shows it, and the admin changes it there (2.3.1+), from MIN_RETENTION_DAYS to 36500 days. Before 2.3.1 it could be changed with SQL only.
  • Multi-tenant: the admin of the master (the piler tenant) sets the retention of each tenant in the tenant editor, Admin > Tenant > Tenant. The admin of a tenant sees it on the retention page, but can't change it.
  • Multi-node: each node has its own piler.tenant_settings, and each worker archives with its own retention_days. Nothing copies it from the master: change it on every worker too, see Change a tenant's settings. The retention page of a multi-node master shows the master's value and doesn't change it. A worker without the tenant's row archives with DEFAULT_RETENTION_DAYS of the piler daemon (3653 days).

The default retention can't be shorter than MIN_RETENTION_DAYS (30 days by default), see Config options.

Retention rules#

A retention rule keeps the emails it matches for a number of days instead of the default retention, e.g. 30 days for newsletters, or 8 years for the emails of the accounting department.

Field Matches
Domain the domain of the From address
Sender (From) the From address
Recipient (To) any recipient: To, Cc, and the envelope recipients
Subject contains the subject
Body contains the first 10 KB of the text of the email
Message size the size of the email in bytes, with >, <, >= or <=
Spam status spam or not spam, by the SPAM_HEADER_LINE of the piler daemon
  • All the filled fields must match. An empty field matches every email.
  • The text fields match a part of the value, regardless of case. example.com in Domain also matches sub.example.com and myexample.com; @example.com in Sender matches only the addresses of that domain.
  • The first matching rule wins, in the order the rules were created. If the rules overlap, e.g. a domain and a sender of that domain, create the more specific one first; there is no other priority.
  • Days: at least MIN_RETENTION_DAYS. A rule shorter than the default retention is saved after the admin types its days, to confirm it.
  • Disable a rule with its switch to stop using it and keep it for later; a deleted rule is gone.

Exclusion rules#

An exclusion rule has the same fields as a retention rule, without the days. An email it matches is not archived: the piler daemon removes it, logs it with status=discarded and the name of the rule, and counts it among the discarded emails. It can't be recovered from piler later, so keep these rules narrow, e.g. the notifications of a monitoring system.

The envelope recipients count for the exclusion rules too: an email with a recipient a rule matches is excluded even if that recipient is only in the envelope.

When a change applies#

  • The piler daemon reads the rules again every 30 minutes: a new or changed rule applies within 30 minutes. It reads the default retention for each email.
  • pilerimport reads the rules and the default retention when it starts: a change applies to the next import run.
  • Multi-node: the rules are created on the master. Each worker copies them from the master every hour with sync_tables_from_master.sh, a cron job of the piler user on the worker: the Docker worker image turns it on, on a native worker check that the crontab of the piler user has it uncommented (crontab -l -u piler). A change reaches the workers within an hour and a half. The legal holds are copied the same way. The default retention isn't copied, see Default retention.

Simulate a rule#

Simulate counts the archived emails a rule matches, with the date of the oldest and the newest. For a retention rule it counts those older than the rule's days: the emails that would be past their retention if the rule had applied when they were archived. It changes nothing, the rules don't apply to the emails already archived.

The simulation matches the criteria like the archiving does (2.3.1+), except the body: the text of the archived emails isn't in the database, so a body criterion isn't counted, and the result says so. A rule with a body criterion may match fewer emails than the simulation shows. Before 2.3.1 the simulation of a rule with a domain, sender, recipient or body criterion failed.

On a multi-node master the emails are on the workers, so the simulation there finds none.

Who can change the rules#

  • Single tenant: the admin.
  • Multi-tenant: each tenant has its own rules, in its own database. The admin of a tenant manages them on the tenant's retention page. With ENABLE_TENANT_RETENTION_POLICIES=false, globally or in a tenant's settings_json, the admin of that tenant can view and simulate the rules, but not change them: the page says the platform operator manages them. To change them, the operator turns the setting on for the tenant, changes the rules as the tenant's admin, and turns it off again. The rules of the piler tenant are always managed by its admin. See Config options.

Every change of a rule is on the Audit Log tab of the retention page, with the rule before and after the change, and in the audit log of the configuration changes (2.3.0+). A change of the default retention on the retention page is in the audit log as retention_default_update, with the previous value.

Purge#

The purge removes the emails whose retention date has passed: their stored files and their index entries, and marks them deleted in the database. It runs every night at 2:02 (/usr/libexec/piler/purge.sh, a cron job of the piler user; in Docker, of the piler container), for each tenant whose enable_purge is on in piler.tenant_settings. It's on by default.

  • Legal hold: an email of a custodian under an active legal hold (a user, a domain or a group, as sender or recipient) is not purged, even after its retention date.
  • Multi-node: the purge runs on every worker, by the enable_purge of the tenant on that worker. A worker without the tenant's row doesn't purge it.
  • Turned off (enable_purge is 0): the emails stay in the archive after their retention date, until it's turned on again.