Importing / exporting emails#
This documentation applies to Piler enterprise edition 1.7.2
Revision #1
Publication date: Apr 20, 2023
Exporting emails#
The GUI allows users to export their emails they can see in the search results. It’s suited to download one page of emails at a time, usually 20, 30, 50 or even 100 depending on the page length set for the given users.
However, occasionally more emails needed to be exported. The pilerexport CLI utility can be used by administrators with shell access to the archive host.
The pilerexport CLI utility allows you to export messages from the archive to EML files to the current directory. Each exported message is written to a distinct RFC-2822 formatted EML file, so expect lots of EML files in the current directory.
IMPORTANT! The pilerexport utility is setuid to the piler user, and to let it create the EML files you have to cd to a directory where the piler user has write access.
usage: pilerexport
[-c <config file>] Config file to use if not the default
-a <start date> Start date in YYYY.MM.DD format
-b <stop date> Stop date in YYYY.MM.DD format
-f <email@address.com> From address
-r <email@address.com> Recipient address
-e <email@address.com> Export both sent and received emails
-F <domain.com> From domain
-R <domain.com> Recipient domain
-s <size> Min. size
-S <size> Max. size
-w <where condition> Where condition to pass to sphinx, eg. "match('@subject: piler')"
-m <max. matches> Max. matches to apply to sphinx query (default: 1000)
-i <index list> Sphinx indices to use (default: main1,dailydelta1,delta1)
-A Export all emails from archive
-W <customer> Customer
-z <zip file> Write exported EML files to a zip file
-Z <batch size> Zip batch size. Valid range: 10-10000, default: 2000
-D <max files> Max. number of files to put in a single directory, default: 2000
-p <private key> Sign the created zip file with <private key>
-E <export dir> Dump emails to this directory
-o Export emails to stdout
-q Quiet, print only errors
-d Dry run
If you want to see what messages piler would export, then use the -d (or -dryrun) command line option. It will print a list of IDs associated with each message.
To prevent you dumping the whole archive you have to specify the from addresses and/or the recipient addresses and/or the start/stop dates. If you really want to export the whole archive, then use the -A (or --all) command line option. You may need lots of disk space and time to do so.
You may specify more email addresses, see the examples below. The conditions are in Boolean AND relation with each other.
A few examples using the customer called “fictive”*:
export all emails sent on 2015.11.24 (aka a daily backup):
pilerexport -W fictive --start-date 2015.11.24 --stop-date 2015.11.24
export all emails sent on 2015.11.24 bigger than 10 kB:
pilerexport -W fictive --start-date 2015.11.24 --stop-date 2015.11.24 -s 10000
export all emails if size is between 100k and 2MB AND the sender is any of aaa@aaa.fu,bbb@aaa.fu AND the recipient is you@example.com:
pilerexport -W fictive --start-date 2015.11.24 --stop-date 2015.11.24 -s 10000 -S 200000 -f aaa@aaa.fu -f bbb@aaa.fu -r you@example.com
export all emails sent to Gmail:
pilerexport -W fictive -R gmail.com
export all emails received from Yahoo:
pilerexport -W fictive -F yahoo.com
export all emails matching the sphinx query “subject: test”:
pilerexport -W fictive -w "match('@subject test')" -i fictive_main1,fictive_dailydelta1
*: For no-tenant installations you don’t have to specify the customer.
Importing emails#
The pilerimport utility allows you to import legacy emails. It supports a wide variety of sources, eg. imap, pop3 accounts, directories with EML files, MBOX files.
IMPORTANT! The pilerimport utility is setuid to the piler user, and to let it create some temporary files you have to cd to a directory where the piler user has write access.
usage: pilerimport
[-c <config file>] Config file to use if not the default
-e <eml file> EML file to import
-m <mailbox file> Mbox file to import
-d <dir> Directory with EML files to import
-i <imap server> IMAP server to connect
-K <pop3 server> POP3 server to connect
-u <username> Username for imap/pop3 import
-p <password> Password for imap/pop3 import
-P <port> Port for imap/pop3 import (default: 143/110
-t <timeout> Timeout in sec for imap/pop3 import
-x <folder1,folder2,....folderN,> Comma separated list of imap folders to skip. Add the trailing comma!
-f <imap folder> IMAP folder name to import
-g <imap folder> Move email after import to this IMAP folder
-b <batch limit> mport only this many emails
-s <start position> Start importing POP3 emails from this position
-W <customer>
-D Dry-run, do not import anything
-o Only download emails for POP3/IMAP import
-O <move dir> Move downloaded only emails to this directory (applies to imap/pop3)
-r Remove imported emails
-y Read emails from export
-q Quiet mode
A few examples using the customer called “fictive”*:
import a single EML file:
pilerimport -W fictive -e message-from-jack.eml
Import a single EML file from STDIN:
cat some.eml | pilerimport -W fictive -e -
Import all messages from a Unix mbox file:
pilerimport -W fictive -m /path/to/202008.mbox
Import all messages from a directory having EML messages (it’s recursive):
pilerimport -W fictive -d /path/to/dir
Import all messages from a directory having EML messages and remove all successfully imported eml files:
pilerimport -W fictive -d /path/to/dir -r
Import messages from a POP3 server:
pilerimport -K pop3.yourdomain.com -u username -p password
Importing from an IMAP server#
Though the pilerimport utility can import emails from imap servers, it might have a hard time with some
imap server implementations. The recommended method is to use the imapfetch.py utility to download the
emails from the imap servers, then run pilerimport -d to process emails in the directory, eg. pilerimport -d /path/to/dir -r
export PATH=$PATH:/usr/libexec/piler
imapfetch.py -s imap.yourdomain.com -u username -p password
pilerimport -W fictive -d . -r
Download emails from the IMAP folders “2019” and “2020” only:
imapfetch.py -s imap.yourdomain.com -u username -p password -f 2019,2020
By default imapfetch.py excludes the following IMAP folders: junk,trash,spam,draft
Skip only the “junk” and “draft” IMAP folders:
imapfetch.py -s imap.yourdomain.com -u username -p password --skip-list junk,draft
How to import a PST file#
The pilerimport can process only EML files directly. However, the readpst utility from the libpst package
can extract the contents of a legacy PST file created with Outlook 2003. Unfortunately this
tool may provide bogus email headers, eg. From: <MAILER-DAEMON>, missing To: address(es),
Date:, Message-ID: headers, and even Received: lines. These are crucial for piler, and you will
have problems without them, so be sure to verify the extracted files before you import them
if these mentioned email headers are fine.
You may use the following command:
readpst -M -b /path/to/file.pst
For PST files created with a more recent version of Outlook try the Bitrecover Pro edition.
Once you have the RFC-2822 formatted EML files extracted from the PST file, import them with the pilerimport command.
How to use impersonation with Exchange#
add-mailboxpermission username -AccessRights FullAccess -user superuser
pilerimport -i mail.domain.com -u domain.com/superuser/username -p superpassword
remove-mailboxpermission username -AccessRights FullAccess -user superuser
Replace 'username' with your email user (no @domain necessary), 'superuser' with your import user, and 'superpassword' with your import user's password, and 'domain.com' with your domain.
Importing IMAP accounts over the GUI#
To enable the import menu in the GUI, set the following in /etc/piler/config-site.php:
$config['ENABLE_IMPORT'] = 1;
For multitenant installations you may enable it selectively for certain customers when putting the above line to a per customer config file. The administrators may go to the administration / import menu, and add an import job by specifying the imap connection type (ssl or not), the imap server name, and the username and password.
Visit https://mailpiler.com/import-over-the-gui/ for screenshots.