Archiving envelope (Bcc) recipients and the envelope sender#

This documentation applies to Piler enterprise edition releases 2.2.3+ (envelope files in pilerimport: 2.2.4+)

Revision #2

Publication date: 2026-OCT-16

A Bcc recipient is not in the To or Cc header of an email, only in the SMTP envelope. An archive that reads the recipients from the message alone doesn't know about them, so a Bcc recipient can't find the email in the archive.

Piler solves this with envelope files: piler-smtp writes the envelope of an email (its sender and its recipients) to a small file next to it, and the piler daemon archives the recipients as recipients of the email. The email itself is stored exactly as it arrived.

The envelope file also has the envelope sender of the email, which auditors can see, see The envelope sender.

For migrated mail, pilerimport reads the same envelope files placed next to the imported EML files, and can add the recipients to mail imported earlier, see Migrated mail.

This page explains how to set it up, what to expect, and the trade-offs behind the design.

How it works#

Mail server                     Archive server
┌──────────┐  RCPT TO:<customer1+anna=customer1.com@archive.example.com>
│ Postfix  │─────────────────>┌────────────┐  email + envelope file  ┌───────┐
│ (or any  │      (SMTP)      │ piler-smtp │────────────────────────>│ piler │
│  MTA)    │                  └────────────┘                         └───────┘
└──────────┘

The mail server sends a copy of each email to the archive. It encodes every original recipient in the archive address it sends the copy to.
piler-smtp decodes the recipients and writes them to an envelope file next to the email.
The piler daemon reads the envelope file together with the email and archives its recipients like the ones in To and Cc.

The archive address#

<archive local part>+<recipient local part>=<recipient domain>@<archive hostname>

RCPT TO:<customer1+anna=customer1.com@archive.example.com>
RCPT TO:<customer1+ben=customer1.com@archive.example.com>

The archive local part (the tenant name in a multi-tenant deployment) ends at the first +, and the last = stands for the @ of the original recipient.

A plain archive address (customer1@archive.example.com) still works as before, it just carries no envelope recipient. The email still gets an envelope file, with its envelope sender.

The envelope file#

piler-smtp writes one envelope file per email and tenant in the worker directory, holding the envelope sender and the recipients of that tenant only:

0/customer1-<filename>        the email, exactly as received
0/customer1-<filename>.env    the envelope sender and recipients

v1
from boss@customer1.com
rcpt anna@customer1.com
rcpt ben@customer1.com

Every email has an envelope file, as every email has an envelope sender. The rcpt lines are only there for the recipients encoded in the archive address.

The file is in place before the email shows up in the worker directory. If it can't be written, piler-smtp rejects the email with a 451 response, so the mail server retries it. The piler daemon removes the envelope file with the email once it's archived.

Setup#

Do the steps in this order. If the mail server starts sending encoded addresses before piler-smtp accepts them, they are rejected with 451 invalid tenant until it does.

1. Upgrade piler-smtp, piler and run the migrations#

You need piler-smtp 1.1.4 to support the envelope sidecar.

Envelope files need a piler daemon that knows about them, support added in version 2.2.3. After upgrading, run the SQL migrations as described in the upgrade guide. Migration 027_message_identity.sql creates a new, empty table; it's instant on an archive of any size.

2. Restrict access to piler-smtp#

This is a requirement, not a recommendation. By default piler-smtp accepts email from anyone. With envelope files enabled it must only be reachable by your own mail servers (or those of your provider, e.g. Google Workspace or Microsoft 365).

Whoever can talk to piler-smtp directly chooses both the email and its envelope recipients. Piler has no way to tell if such recipients are real, it trusts what the mail server says.

Use a firewall rule, or the access list of piler-smtp:

# /etc/piler/piler.conf
smtp_access_list=1

# /etc/piler/smtp.acl
# network/prefix permit|reject
192.0.2.10/32 permit
192.0.2.11/32 permit

A client that matches no line is denied.

3. Enable envelope files in piler-smtp#

# /etc/piler/piler.conf
enable_envelope_sidecar=1

Restart piler-smtp. The default is 0: encoded addresses are not accepted, and no envelope files are written.

This one setting covers both the envelope recipients and the envelope sender.

4. Configure the mail server#

For other mail servers see Mail server integration, e.g. Axigen.

Example for Postfix. A recipient_bcc_maps lookup is evaluated for each envelope recipient, Bcc recipients included, so it can do the encoding:

# /etc/postfix/main.cf
recipient_bcc_maps = pcre:/etc/postfix/bcc_recipient_map
sender_bcc_maps = pcre:/etc/postfix/bcc_sender_map

# /etc/postfix/bcc_recipient_map
/^(.+)@(customer1\.com)$/ customer1+$1=$2@archive.example.com
/^(.+)@(customer2\.com)$/ customer2+$1=$2@archive.example.com

# /etc/postfix/bcc_sender_map
/^(.+)@customer1\.com$/ customer1@archive.example.com
/^(.+)@customer2\.com$/ customer2@archive.example.com

sender_bcc_maps keeps the plain archive address. It's what archives the outgoing emails.

Both maps are needed, each covers emails the other one misses:

Email	`recipient_bcc_maps`	`sender_bcc_maps`
`boss@customer1.com` to `someone@gmail.com`	No match, no copy	Copy for customer1
`boss@customer1.com` to `anna@customer2.com`	Copy for customer2 only	Copy for customer1
`boss@customer1.com` to `anna@customer1.com`	Copy for customer1	Same tenant again, archived once
`someone@gmail.com` to `anna@customer1.com`	Copy for customer1	No match

In a single-tenant deployment use your archive local part instead of the tenant names, with one line per mail domain.

piler-smtp accepts several recipients and several tenants in one SMTP transaction, so there is no need for smtp_destination_recipient_limit = 1. With that setting each recipient arrives as a separate copy of the email with its own envelope file. That works too, see below.

Reload Postfix after creating the maps:

postfix reload

5. Verify#

Send an email with a Bcc recipient through the mail server, then check the log of the piler daemon:

/var/piler/tmp/0/customer1-4F2A91C2 has 2 envelope recipients

The Bcc recipient should now find the email in the archive when logged in.

What to expect#

Envelope recipients are recipients. They can find the email in the archive, and they count for the exclusion rules, the retention rules and the direction (internal, inbound, outbound) of the email, like To and Cc recipients do.
The stored email is not modified. The envelope recipients are kept in the database and in the search index, not in the email. A downloaded or exported email is byte for byte what the mail server sent, without the Bcc recipients.
Reindexing keeps them. The envelope recipients are restored from the database when an email is reindexed.
Recipients come from the envelope only. Piler doesn't take recipients from a header inside the email, such as X-Envelope-To. A header is part of the email, so its sender could have put it there.
Invalid addresses are skipped. An envelope recipient that is not a local@domain address is ignored, with a warning in the log.
An unreadable envelope file stops the email. The email is moved to the error directory together with its envelope file, rather than archived without its recipients.

The envelope sender#

The sender in the From header of an email and the sender in the SMTP envelope (MAIL FROM) are often not the same. Bulk mail and forwarded mail have another envelope sender, and so does an email with a forged From header. In a phishing or spoofing investigation it's the envelope sender that tells where an email came from.

With envelope files enabled, piler stores the envelope sender of each email, and shows it to auditors in the message view, under the From line. No other role sees it.

A warning is shown if it differs. If the envelope sender is not the address in the From header, the line has a warning next to it. That is normal for newsletters, mailing lists and forwarded mail, so it's a pointer for the auditor, not a verdict.
A bounce has an empty envelope sender, shown as "empty (bounce)". The warning is shown for it too.
It's only a record. It is what the sending server claimed, nothing verified it. It doesn't decide who can see an email, and the exclusion and retention rules don't use it.
Unusual characters are replaced. Only printable ASCII is stored. Any other byte in the envelope sender is stored as ?, so an address like boss??@example.com means the original had characters that are not ASCII.
It's that of the first copy. If the same email arrives more than once, the envelope sender of the first copy is kept.
Emails archived before the upgrade have none, and no line is shown for them.

Searching by the envelope sender#

Auditors can find emails by their envelope sender:

in the advanced search, with the Envelope sender field;
in the search box, with the mailfrom: label, e.g. mailfrom:bounces@lists.example.org subject:invoice. The label can be anywhere in the search, e.g. invoice mailfrom:bounces@lists.example.org.

It can be combined with the other search terms and filters. A few things to know:

The whole address has to be given. It's an exact match (upper and lower case don't matter), there are no wildcards and no search by domain.
mailfrom:<> finds the bounces, the emails with an empty envelope sender.
Only emails archived with envelope files are found. Older emails have no envelope sender on record.
For an envelope sender with very many emails, the 10000 most recently archived ones are searched, within the date range of the search. If there are more, the search says so: narrow the date range to search the others.
In a multi-node setup the master node asks every worker node for the emails of the envelope sender, the 10000 most recent ones on each. All nodes have to run a version with this feature: the search reports an error, rather than incomplete results, if a worker can't answer.
Other roles can't search by the envelope sender, such a search finds nothing for them.

What piler stores is the MAIL FROM of the transaction with piler-smtp. Postfix keeps the envelope sender of the original email for the copies its bcc maps make. A relay that sends the archive copy with a return path of its own has to pass the original one on. With journaling (e.g. Microsoft 365) it's the address of the journaling system, not of the original sender.

The same email arriving more than once#

Piler archives an email once, its Message-ID identifies it. With envelope recipients the same email often arrives several times, each time with other recipients:

the mail server forwards it to the archive in one SMTP transaction per recipient, or
the sender's server delivered it to your mail server in one transaction per recipient, which is common for Bcc recipients. These are separate emails in your mail server's queue, each with its own Received header.

Dropping the later copies as duplicates would drop their recipients as well. So piler adds the envelope recipients of a later copy to the archived email, if two conditions are met.

1. The copy really is the same email#

A matching Message-ID is not enough. The Message-ID of an email is not a secret: every recipient sees it, and so does everyone who later gets a reply to it. If it were enough, an employee added late to a thread could send themselves an email with the Message-ID of an earlier message, and become its recipient in the archive.

So the copy has to be either identical to the archived email byte for byte, or the same in everything that matters:

the body, with all its attachments,
the From, Sender, To, Cc, Subject and Date headers,
the Content-Type, Content-Transfer-Encoding and Content-Disposition headers.

Other headers may differ. Copies delivered in separate transactions have different Received headers, and they are still recognized.

2. The copy arrives in time#

The copies of an email arrive seconds or minutes after each other, hours if the mail server has to retry. A copy that arrives much later is more likely someone sending the email again.

ENVELOPE_MERGE_WINDOW_HOURS is how long after an email was archived a copy may add its envelope recipients. It's an environment variable of the piler daemon:

Value	Meaning
`24` (default)	Copies arriving within 24 hours add their recipients
a larger number	For mail servers that may deliver a deferred copy later, e.g. `120` for the 5 days Postfix retries by default
`0`	Later copies never add recipients

The window applies to copies arriving by SMTP, and to duplicates in a normal pilerimport run. pilerimport -add-recipients has no window, see Migrated mail.

What happens in each case#

A later copy that...	Result
is the same email, arrives in time, has a new recipient	The recipient is added to the archived email and can find it
is the same email, has no new recipient	Dropped as a duplicate, nothing changes
arrives after the merge window	Dropped as a duplicate, its recipients are not added
has the same `Message-ID`, but is a different email	Dropped as a duplicate, its recipients are not added. With `ARCHIVE_MESSAGE_VARIANTS=true` it's archived as an email of its own instead, see below
has no envelope file	Dropped as a duplicate, as before

When recipients are not added, the piler daemon logs a warning with the Message-ID:

the duplicate arrived 25h0m0s after the archived message, its envelope recipients are not added
the duplicate differs from the archived message, its envelope recipients are not added

The later copy itself is never stored. The archive keeps the first copy of an email.

Emails with the same Message-ID but different content#

By default piler identifies an email by its Message-ID: once an email is archived, any later one with the same Message-ID is a duplicate and is dropped, also if its content is different.

That is not always what you want. Some mail gateways change an email on its way: they rewrite the links in it, or add a disclaimer. If one copy of an email is archived before such a change and another one after it, the two have the same Message-ID and different bodies. The second one is dropped, and with it the recipients that only came with it, for example the members of a distribution list that is expanded late.

With ARCHIVE_MESSAGE_VARIANTS=true piler identifies an email by its Message-ID and its content. An email that has the Message-ID of an archived one but different content is archived as an email of its own, with its own recipients.

Enabling it#

ARCHIVE_MESSAGE_VARIANTS is an environment variable of the piler daemon. The default is false.

ARCHIVE_MESSAGE_VARIANTS=true

Only the piler daemon reads it. It applies to the emails that arrive through piler-smtp. The command line tools ignore it: pilerimport always treats an email with an archived Message-ID as a duplicate, so an import or a re-import never creates a second entry.
In a multi-node setup set it for the piler daemon of every worker node.
It's one setting for the daemon, not for a tenant: it applies to all the tenants the daemon processes.
It needs the database migrations of 2.2.3 (027_message_identity.sql). Without them the setting has no effect, and the piler daemon logs a warning.

What happens to an email with an archived Message-ID#

The email...	Result
is the same as an archived one (see above)	A duplicate. Its envelope recipients are added to that archived email, within the merge window
has different content	Archived as an email of its own
has different content, and the archived one is from before the upgrade to 2.2.3	A duplicate, dropped. See below
is byte for byte the same as a purged one whose metadata was kept	The purged email is restored
has no `Message-ID`	Not affected. Piler gives such an email an id made from its content

There is no limit on the number of emails with the same Message-ID.

Emails archived before the upgrade#

An email archived before the upgrade to 2.2.3 never gets a second entry. Piler has no record of the content of such an email to compare with, so it can't tell if a later one with its Message-ID is the same email with other headers or a different one. It treats it as a duplicate, as before.

This matters for migrations: sending or importing emails again over an archive that already has them can't double them.

What to expect#

More than one entry for a Message-ID. An auditor's search returns each of them. A user finds the ones they are a recipient of. In the thread view each is an entry of its own.
Auditors see which entries belong together. When an auditor opens an email, the other archived emails with its Message-ID are listed as Related copies under the From line, each with its date and size, and a click opens it. In a multi-node setup the list has the copies on all the nodes. If the list could not be put together completely (a node did not answer, or there are more than 100 on a node), it says so. Other roles are not shown this list.
Each entry stands for itself. It has its own recipients, envelope sender and retention, and it's purged on its own. A legal hold on a recipient holds the entries that recipient is on: a hold on a distribution-list member holds the copy that member got, and not the original, unless the sender or a recipient of the original is under hold as well.
More storage if a gateway changes the body for every recipient. With a link or a disclaimer of their own for each recipient, such an email is stored once for each of them. The attachments are still stored once. This is why the setting is off by default: check what your gateway does before enabling it.
In a multi-node setup each node looks at its own archive. The same email arriving on two nodes is archived on both, with or without this setting.
An email that is the same but late is not affected by this setting: if it arrives after the merge window, it's dropped with its recipients.

Which settings for which mail flow#

How your mail server hands an email with several recipients to the archive decides which settings you need. All three cases need enable_envelope_sidecar=1 in piler-smtp and the recipients encoded in the archive address.

How the email arrives	Result	Settings
One copy for all the recipients (one SMTP transaction)	One archived email with all the recipients	Nothing more
One copy for each recipient, the copies are the same (other `Received` headers are fine)	One archived email. The first copy is stored, the others are duplicates and their recipients are added to it, also if the copies arrive at the same moment	`ENVELOPE_MERGE_WINDOW_HOURS` (default 24): raise it only if a copy may be delivered later than that. Leave `ARCHIVE_MESSAGE_VARIANTS` off
One copy for each recipient, with the same `Message-ID` but different bodies (e.g. links rewritten for each recipient)	By default: one archived email, the other copies are dropped with their recipients. With `ARCHIVE_MESSAGE_VARIANTS=true`: one archived email for each copy, each with its recipient	`ARCHIVE_MESSAGE_VARIANTS=true` for the piler daemon

In the third case each recipient finds the copy they got, auditors see the copies listed as related copies, and the body is stored once for each copy (the attachments once).

In a multi-node setup each node decides for itself: the copies of an email only end up as one archived email if they reach the same node. ARCHIVE_MESSAGE_VARIANTS has to be set on every worker node.

If you don't know which case is yours, compare the bodies of two copies of the same email as your mail server sends them to the archive.

Migrated mail (pilerimport)#

When you migrate an archive (e.g. from Mimecast or MailArchiva) with pilerimport, the Bcc recipients are in no header of the EML files either. If you know the mailboxes of each message, write them to an envelope file next to the EML, in the same format piler-smtp writes:

emails/message1.eml
emails/message1.eml.env

v1
rcpt anna@customer1.com
rcpt ben@customer1.com

The from line is optional. Then import as usual:

pilerimport -dir emails -tenantid customer1 -workers 8

The recipients of the envelope file are archived like the To and Cc recipients. -dir skips the .env files, and -remove removes them together with their EML. An EML without an envelope file is imported with its header recipients, as before.

Upgrading from 2.2.3 or earlier: don't put .env files in an import directory before upgrading pilerimport: earlier versions import them as emails.

Adding the recipients to mail imported earlier#

If the mail is already imported without its Bcc recipients, importing it again doesn't help: the emails are duplicates, and the archived emails were archived long before the merge window. Use -add-recipients instead: it imports and stores nothing, it only adds the recipients of the envelope files to the archived emails.

First see what would happen, nothing is changed:

pilerimport -dir emails -tenantid customer1 -add-recipients -dry-run

Then run it:

pilerimport -dir emails -tenantid customer1 -add-recipients -workers 8

The summary counts each EML:

Result	Meaning
Messages updated	the archived email got recipients
Already complete	it had all the recipients already
Not archived	no archived email has its Message-ID
Not a copy	archived emails have its Message-ID, but none is the same email: nothing is added
No envelope file	the EML has no envelope file, or no valid recipient in it

There is no time window here: the files come from you, not from the network, so nobody can send an email again to become its recipient. But the first rule still applies: an archived email only gets recipients from an EML that is the same email. For emails imported before migration 027_message_identity.sql, only a byte for byte identical file is the same email. Use the files you imported, not a new export from the old archive: an export may differ in line endings or header order, and then the result is "not a copy".

Audit trail. Every email that gets recipients has an entry in the audit log, with the action recipients, the user who ran pilerimport (<login>@<host> (pilerimport)), the file and the recipients added. The entry is written before the recipients are added. If it can't be written, nothing is added and pilerimport stops, so no mailbox gets access to archived mail without a trace.

What is not changed. The direction, the retention date and the exclusion rules of the email are not evaluated again, as with the copies above.

Trade-offs and limitations#

The rules above are deliberately strict. An archive that adds a recipient to an email too easily gives access to the wrong person, or records a recipient the email never had. The price is that in some cases a real recipient is not added.

A copy that was modified on the way is not recognized. If something changes the body or the headers listed above between two copies (a mailing list footer, a disclaimer added by a gateway, a [SPAM] tag in the subject, a re-encoded body), the later copy is a different email as far as piler can tell. Its envelope recipients are not added. Make sure such modifications happen either before the email is copied to the archive for every recipient, or not at all. With ARCHIVE_MESSAGE_VARIANTS=true such a copy is archived as an email of its own instead, see above.

A copy that arrives after the merge window is not merged. If your mail server may hold a copy for longer than a day, raise ENVELOPE_MERGE_WINDOW_HOURS. A longer window is more forgiving for late deliveries and gives more room for the replay described below.

Rejected copies are not kept. A copy whose recipients are not added is removed like any other duplicate. The warning in the log is the only trace, so watch for it after enabling the feature.

An email can be sent again within the merge window. Someone who has an email can send it again unchanged, to an address of their choice, while the merge window is open. That address is then recorded as a recipient of the original email. The sender of such a copy learns nothing new, they already had the email, and they could have forwarded it. But the archive then lists a recipient the original sender did not address. The merge window limits this to the first day, it does not prevent it.

Emails with an invalid or missing Date header. Piler then takes the date from the first Received header, which differs between copies. For these emails only a byte for byte identical copy adds its recipients.

Emails archived before the upgrade. For an email archived before migration 027_message_identity.sql was run, only a byte for byte identical copy adds its recipients. Copies arrive shortly after each other, so this only matters for the first hours after the upgrade.

Direction and retention are those of the first copy. A later copy with an external recipient doesn't turn an internal email into an outgoing one, and a retention rule that would only match a later recipient doesn't apply. The recipients added later are not counted in the sender and recipient statistics either.

Storage. Piler keeps a 32 byte digest for each archived email to recognize its copies, about 6-7 GB for 100 million emails. It's removed when the email is purged.

Troubleshooting#

The mail server gets 451 invalid tenant. enable_envelope_sidecar is not enabled in piler-smtp (or it was not restarted), and the encoded address is taken for a tenant name.

A Bcc recipient can't find an email. Check in this order:

The mail server sent the copy to the encoded address: look for customer1+anna=customer1.com@archive.example.com in its log.
The piler daemon saw the envelope file: ... has N envelope recipients in its log.
There is no warning for the email: search the log for envelope recipients are not added and for the Message-ID of the email.
The email arrived more than once, and the copy of this recipient came after the merge window or was modified on the way, see above.

Skipped N invalid recipients. The envelope file had recipients that are not valid addresses. Check the maps of the mail server, most likely the encoding of an address went wrong.

.env files are left in the worker directory. An envelope file is removed with its email. One that has no email (e.g. after a crash) is removed by the piler daemon after 10 minutes. A piler daemon from before this feature doesn't know what these files are: don't enable enable_envelope_sidecar before upgrading it.