Skip to content

Import emails#

This documentation applies to Piler enterprise edition 2.1.0 (envelope files and -add-recipients: 2.2.4+, PST files and the Microsoft 365 download options: 2.3.0+)

Revision #5

Publication date: 2026-OCT-30

Import local emails#

Usage: pilerimport [options]

Options:
  -add-recipients
        Don't import: add the recipients of the envelope file next to each email (<email>.env) to the archived copy of the email, however long ago it was archived
  -dir string
        Process all files in directory (recursively)
  -dry-run
        With -add-recipients: report what would be added, change nothing
  -eml string
        Process a single EML file
  -mbox string
        Process an mbox file (extracts all messages)
  -progress
        Show progress bar (suppresses info logs)
  -pst string
        Process a PST file (Outlook, Enterprise Vault export): its mail, not appointments, contacts...
  -remove
        Remove file after successful processing
  -skip-deleted
        With -pst: don't archive the items of Deleted Items
  -tenantid string
        Tenant ID (overrides TENANT_ID env var)
  -workers int
        Number of parallel workers (default 4)
  -x500-map string
        With -pst: CSV file of legacyExchangeDN,mail to replace Exchange X.500 addresses with SMTP addresses

Examples:
  pilerimport -eml message.eml -tenantid fictive
  pilerimport -dir /path/to/emails -tenantid fictive -workers 8
  pilerimport -mbox archive.mbox -tenantid fictive -workers 4 -progress
  pilerimport -pst mailbox.pst -x500-map dn-to-mail.csv -workers 4 -progress
  pilerimport -dir /path/to/emails -remove -workers 4 -progress
  pilerimport -dir /path/to/emails -add-recipients -dry-run

Bcc recipients of migrated mail#

A Bcc recipient is in no header of an EML file, so it can't find the email after the import. If you know the recipients of each email, put them in an envelope file next to the EML (message.eml.env for message.eml):

v1
rcpt anna@customer1.com
rcpt ben@customer1.com

pilerimport archives these recipients like the To and Cc ones. For emails that are already imported, -add-recipients adds them without importing the emails again. See Migrated mail for the details, and the audit log entries it writes.

Import from an IMAP server#

Usage of imapfetch:
  -batchsize int
        batch size to download before processing
  -dir string
        directory to chdir
  -folders string
        Comma separated list of IMAP folders to download
  -i    Read IMAP conn data from import table
  -j int
        job ID
  -no-ssl
        Do not use SSL/TLS
  -oauth2-token string
        OAuth2 access token file
  -password string
        IMAP password
  -port int
        port number (default 993)
  -print-folders
        only print IMAP folders, then exit
  -readonly
        Open mailbox in read-only mode
  -remove
        remove downloaded messages
  -search string
        Search expression
  -server string
        IMAP server
  -tenantid string
        tenant ID (required)
  -user string
        IMAP user
  -verbose
        verbose mode
  -x string
        IMAP folders to skip (default "junk,trash,spam,draft")

Examples:
  imapfetch -server imap.aaa.fu -user bela@aaa.fu -password abcde123 -tenantid piler

The imapfetch utility can download emails from an IMAP server to the current directory, then you can use pilerimport to process them.

pilerimport -dir . -remove

Import from Google Workspace#

Enable Gmail API and Admin SDK API#

Go to Google Cloud Console

Create a project, eg. “import-emails”

Click on “APIs & Services”, then “Enable APIs and Services”

Enable the “Gmail API” and “Admin SDK API”

Create a Service Account#

Click on “APIs & Services”, then “Create credentials”, and select “Service account”

After creating it, click on the service account.

Select the “Keys” tab, click on “Add key” / “Create new key”. The key type shall be “JSON”, then save the json file.

Grant domain wide delegation#

We are going to impersonate users. To do that go Google Admin Console

Click on “Security”, then “Access and data control” and “API Controls”.

Then select “Manage Domain Wide Delegation”

Add the Service Account Client ID and authorize the following scopes:

https://www.googleapis.com/auth/gmail.readonly
https://www.googleapis.com/auth/admin.directory.user.readonly

Download all emails#

Use your Google Workspace admin email (eg. admin@example.com) with the json credentials file you created above:

download-emails-google-workspace.py --admin admin@example.com --credentials import-emails-d25d9743f508.json

The script downloads emails to “emails/user@example.com” directory.

Then all you need to do is to import them:

chown -R piler:piler .
pilerimport -dir emails -remove

Import from Microsoft 365#

Create a new application with a secret in Azure portal#

Grant the following application type permissions:

  • Mail.Read
  • User.ReadBasic.All

If everything is set properly, you see a similar output for the decoded access token. Notice the “roles” section.

{
    "aud": "https://graph.microsoft.com",
    "iss": "https://sts.windows.net/fe3b7b62-7d1a-4bcc-a513-e3b0798e601f/",
    "iat": 1739172947,
    "nbf": 1739172947,
    "exp": 1739176847,
    "aio": "k2RgYGjT0svi6ArYtthhkX+BhEkbAA==",
    "app_displayname": "imap-import",
    "appid": "4bc3a597-1602-4d3f-b07e-264374805380",
    "appidacr": "1",
    "idp": "https://sts.windows.net/fe3b7b62-7d1a-4bcc-a513-e3b0798e601f/",
    "idtyp": "app",
    "oid": "456bad53-c1db-4ab2-b149-5c95e47c556c",
    "rh": "1.AVIAYns7_hp9zEulE-OweY5gHwMAAAAAAAAAwAAAAAAAAAC6AABSAA.",
    "roles": [
        "User.ReadBasic.All",
        "Mail.Read"
    ],
    "sub": "456bad53-c1db-4ab2-b149-5c95e47c556c",
    "tenant_region_scope": "NA",
    "tid": "fe3b7b62-7d1a-4bcc-a513-e3b0798e601f",
    "uti": "5hddMVdLBEiufY9lQN7YAA",
    "ver": "1.0",
    "wids": [
        "0997a1d0-0d1d-4acb-b408-d5ca73121e90"
    ],
    "xms_idrel": "7 10",
    "xms_tcdt": 1669093587
}

Install the Python packages#

The script needs Python 3.9 or newer, with the requests and msal packages. Recent distributions refuse a system-wide pip install, so install them in a virtual environment:

python3 -m venv ~/o365-import
~/o365-import/bin/pip install requests msal

Create config.py#

Copy the script to a working directory (the packages install it as /usr/libexec/piler/download-emails-o365.py), and create config.py next to it. Set the values for client_id, client_secret and tenant_id. Feel free to customise the list of SKIP_FOLDERS: the mailbox folders that won't be downloaded.

config.py:

DEBUG = False
SKIP_FOLDERS = ['Deleted Items', 'Drafts', 'Junk Email']
client_id = ".........."
client_secret = "......"
tenant_id = "......."

Usage#

Download the emails of a single mailbox:

~/o365-import/bin/python download-emails-o365.py --email user@example.com

Or every user's emails, by leaving out --email. Mailboxes the app may not read (when an application access policy limits it to some mailboxes) and users without a mailbox are skipped and listed in the summary.

Options (2.3.0+):

Option
--email Download this mailbox only (default: every mailbox)
--before YYYY-MM-DD Only emails received before this day (UTC), e.g. when journaling started: the archive has the later ones
--dir Download the emails here (default: emails in the current directory)

Each mailbox gets its own directory, named by the user's id, e.g. 8f7cd945-c816-491b-8c8c-577bd664189a, with the folder structure of the mailbox:

ls -la emails/8f7cd945-c816-491b-8c8c-577bd664189a

total 12
drwxr-xr-x 2 piler piler 4096 May 30 09:29  Inbox/
drwxr-xr-x 3 piler piler 4096 May 30 09:29  folder2/
drwxr-xr-x 2 piler piler 4096 May 30 09:29 'test folder 1'/

Each email is written to <dir>.tmp first (emails.tmp) and moved into place when it's complete, so the download directory never has a half written email. <dir>.tmp must be on the same filesystem: if --dir is a mount point, use a directory inside it.

Throttling, interruptions and the exit code#

Microsoft 365 throttles large downloads. The script waits as long as Microsoft asks and tries again, also after network errors. A large tenant can take hours or days.

If the run is interrupted, or ends with failures, run the same command again: it skips the emails already in the download directory and downloads only the missing ones. The summary at the end lists what was downloaded, skipped and failed, per mailbox and in total. The exit code:

Exit code
0 Complete
1 Some emails couldn't be downloaded: run the script again
2 Complete, except emails Microsoft 365 can't export (logged by id): running it again won't get them

Import the emails#

When the download is complete (exit code 0 or 2), run pilerimport to import the emails. The command below processes all emails in the “emails” directory, and removes the processed emails:

chown -R piler:piler emails
pilerimport -dir emails -remove

Import only after the download is complete: the script knows what it downloaded by the files in the download directory, so after pilerimport -remove a new run would download everything again.

Import PST files#

From 2.3.0 pilerimport imports PST files (Outlook data files) directly, no conversion step is needed. This works with PST files exported from classic Outlook, Exchange or Microsoft 365, and with Enterprise Vault exports, in both the current (Unicode) and the old (ANSI, Outlook 97-2002) format.

pilerimport -pst mailbox.pst -workers 4 -progress

Installed setuid, pilerimport reads the PST as the piler user, so make it readable for piler first:

chown piler:piler mailbox.pst

What is archived#

  • All mail, including Sent Items, Drafts and meeting requests
  • Received mail with its original headers
  • Sent mail and drafts with headers built from the PST: the sender and the recipients with their SMTP addresses, Bcc recipients included, so the sender's copy of a Bcc'd email can be found by every recipient
  • The plain text and HTML bodies (rich text bodies are converted), attachments with their file names and types, inline images, and messages forwarded as attachments
  • S/MIME signed and encrypted emails as they were sent, so signatures stay verifiable

Appointments, contacts, tasks, notes and Enterprise Vault shortcuts are not email: they are counted and skipped. With -skip-deleted the items of the Deleted Items folder are skipped too.

Importing the same PST file again archives nothing new, its emails are duplicates. -remove doesn't work with -pst: a PST has items that are not archived, so pilerimport never removes it.

Exchange addresses#

Email sent within an Exchange organization may have no SMTP address in the PST for internal senders and recipients, only an Exchange address like /O=EXAMPLE/OU=EXCHANGE ADMINISTRATIVE GROUP (FYDIBOHF23SPDLT)/CN=RECIPIENTS/CN=.... A regular user doesn't see those emails, since their address is not in them. Export the Exchange addresses of the users with their SMTP addresses from Active Directory:

Get-ADUser -Filter * -Properties legacyExchangeDN,mail |
  Select-Object legacyExchangeDN,mail | Export-Csv dn-to-mail.csv

and give the file to pilerimport:

pilerimport -pst mailbox.pst -x500-map dn-to-mail.csv -workers 4 -progress

pilerimport replaces the Exchange addresses with the SMTP addresses before archiving the emails. If users had other Exchange addresses earlier (the X500: entries of their proxyAddresses), add them to the file as further rows with the same SMTP address. A file that can't be read stops pilerimport before it archives anything.

Summary#

At the end pilerimport prints what it did with the items of the PST:

=== Import Summary ===
Tenant ID:       piler
Total files:     2178
Successful:      2165
Duplicates:      13
Excluded:        0
Failed:          0
PST items:       2271
Not mail:        93 (appointments, contacts, tasks...; skipped)
EV shortcuts:    0 (skipped)
X.500 replaced:  412 addresses
Database:        piler
Manticore Index: piler

Attachments that can't be archived (attachments by reference, OLE objects) are counted and logged; their email is archived without them.

Converting PST files to EML#

To convert PST files to EML files without importing them, for example to check them first or to import them elsewhere, use the pst2eml tool. It's the converter pilerimport uses, as a standalone program for Linux, Windows and macOS:

pst2eml -o out/ mailbox.pst
pilerimport -dir out/

Without the -x500-map step, this suits PST files of received mail and of organizations without Exchange.