Administering piler enterprise edition#

This documentation applies to Piler enterprise edition 1.7.3

Revision #1

Publication date: Jul 15, 2023

Prerequisites#

You have installed Piler enterprise using either the multiple nodes layout or the single node layout. The Piler GUI uses the http hostname to determine whether it’s the main admin site for the provider, or the customer site for customer admin and customer end users.

This document applies to the main admin site for the provider. In the following examples gui-int.aaa.fu is used as the main admin site URL.

You are free to configure nginx on the GUI node to use the internal network, though it’s up to you, if you want access to the main admin site from the Internet. It’s also best to set up https for security reasons.

Using the GUI at the first time#

Piler features a web based GUI that can be accessible with a browser. Mobile devices are not supported officially at the moment.

Log in as administrator using the following account:

admin@local:pilerrocks

Then go to the user menu at administration / users menu, and change the admin password immediately. It’s also recommended to remove the auditor user, since it’s not used in the main admin site.

If you want to use https on the GUI, then edit config-site.php, and make sure you set https in the SITE_URL variable:

$config['SITE_URL'] = 'https://piler.yourdomain.com/';

Adding customers#

Providers have several domains and mailboxes. Piler uses the concept of customer to distinguish between organizations. For Piler a customer is an organization with 1 or more domains, and several mailboxes. Piler separates customers on the disk level and on the database level. Each customer has its own store directory, sphinx database and mysql database.

Let’s say you want to archive emails for Fictive Company Co. Before doing so you have to create this company in piler, ie. you have to create a customer.

Create domains for the customer#

Before creating the customer itself, you have to create the domains for this company. Let’s say Fictive Company Co. has three domains:

So go to the administration / domains page in the GUI, and set these three domains one per line in the Domains field, and set fictive.com to the mapped domain as well. The mapped domain identifies the primary domain for the company.

Then click on Add, and verify that these domains are added.

Hint: if you need support for domain names having characters other than the English alphabet, you have to set the appropriate $config['DOMAIN_REGEX'] value in config-site.php. See the default value in config.php.

Creating the customer#

After creating domains for the customer, go to administration / customer page, then select the primary domain for the customer, add a customer ID, eg. fictiveco for the customer, fill the GUI URL field as well, then click Add, and verify that the customer is created.

The GUI checks the hostname part of the URL (HTTP_HOST) to figure out which customer site the user visits. So each customer must have their own dedicated URL. The GUI URL field must be only the hostname (eg. fictiveco.myarchive.domain), no http:// prefix, etc.

If you set $config[‘USE_SMTP_GATEWAY’] = 1; in config-site.php, then the customer creating form supplies you with an uuid-like address for the ‘Archiving address’ field, eg. 5cc03cf3-2a32-0d7e-02b8-7b29882e15dc@smtp.aaa.fu. You may accept it or you may change it, either way make sure that to rewrite 5cc03cf3-2a32-0d7e-02b8-7b29882e15dc@smtp.aaa.fu to @workernodename.aaa.fu on the smtp gateway, because the piler-smtp daemon only accepts the customer id as the local part of the rcpt to address.

When the page refreshes you’ll notice that the customer is in PENDING state, because it takes time to create a tenant for the new customer. The Piler cron jobs create store directories, a mysql database, and a sphinx database for this customer on the GUI node and on the worker nodes.

The latter has a challenge: we have to restart the sphinx daemon, and the daemon needs a certain time while it reads and caches indices, so the restart happens at 00 minute, once in every hour.

So please wait sending emails for the new customer until the customer status is OK.

Warning! If the status is OK on the GUI, it still takes up to an additional hour until the customer is created on the worker nodes as well.

You may send all emails for fictiveco to fictiveco@smtp.aaa.fu address. The piler smtp daemon checks the local part of the address to determine that the email belongs to which customer. If it’s unknown then the message is rejected.

How to script adding customers#

Check out the REST API docs.

Quotas#

Piler is not able to handle any customer quota. Whatever email it receives, it also archives it - provided that it’s for a valid customer. The recommended method is to regularly check the customer_settings table in the piler database, and set alerts in case any of your customer’s used column becomes greater than the quota value.

Note that you are free to fix the SQL query in the customer.cf file on the smtp gateway to include a check for the quota column as well, eg.

query = SELECT 'OK' FROM customer_settings WHERE customer_address='%s' AND enabled=1 AND (quota=0 OR quota > used)

However, if a customer has reached their quota, then the smtp gateway will encounter an internal error as well as drop all of their incoming emails which may not be a good idea. So be sure to carefully decide if you really want this behaviour enabled.

You may check the current usage of all customers on the Administration / customer page. Note that the raw email size is used for the calculations, not the actual stored sizes. The values are in GBs.

Note that per domain or per user quotas are not supported.

After the customer is created, you should configure the GUI for the customer, see the next chapter in the manual.

Configuring the GUI#

The GUI ships a default config file /var/piler/www/config.php. You should NOT modify this file! You should put all the site related customized settings to /etc/piler/config-site.php.

It’s also possible to create customer specific settings (eg. authentication details) for the GUI. To do this, cd /etc/piler/sites and create config-site-.php, eg. config-site-fictiveco.php. When customer fictiveco visits the GUI, then the settings in config-site-fictiveco.php override existing values.

Customers should visit the GUI URL you configure when creating a new customer. It can be a subdomain in your domain (eg. https://fictive.aaa.fu/ ), or can be a subdomain in the customer’s domain. eg. https://archive.fictive.com/ to access their own archive.

In the customer specific config you may set several settings for the customer, eg. how to authenticate the customer’s users. See the User authentication methods in the documentation.

User authentication methods#

The piler GUI supports several methods for authenticating users.

Authenticating against the local piler database#

Normally only special users exist in the local piler database, eg. admin@local, auditor@local, etc. and the regular users are queried from an external database, eg. LDAP, AD, IMAP, etc.

It's possible to add the regular users manually, however it’s recommended as the last resort when a central user database is not available.

Note that these settings should usually go to the customer specific config file, eg. config-site-fictiveco.php.

Authenticating against an LDAP server#

This is the preferred way to authenticate regular users. The details depend on what mail system and the LDAP server the customer has. In general you have to enable the LDAP authentication (see the following paragraph), then create a helper account that can query the LDAP directory.

To enable LDAP authentication, set the following in the customer’s site config, eg. config-site-fictiveco.php (for notenant installations write it to /etc/piler/config-site.php):

$config['ENABLE_LDAP_AUTH'] = 1;

Then go to the main admin site to administration / ldap menu, and create an ldap entry for the used LDAP server. You have to set the ldap hostname, ldap base DN where to limit the scope of the ldap query, also the ldap helper account parameters. When you are done, you may click on the test button, and the gui tries to login to the ldap host using the helper account.

Before finishing you have to set a few ldap parameters, and specify the ldap attribute storing the email addresses, name of the user object class, name of the distribution list object class and its attribute name.

Active Directory#

LDAP_ACCOUNT_OBJECTCLASS = 'user';
LDAP_DISTRIBUTIONLIST_OBJECTCLASS = 'group';
LDAP_DISTRIBUTIONLIST_ATTR = 'member';

iredmail#

LDAP_HELPER_DN = 'cn=vmailadmin,dc=yourdomain,dc=com';
LDAP_BASE_DN = 'o=domains,dc=yourdomain,dc=com';
LDAP_ACCOUNT_OBJECTCLASS = 'mailUser';
LDAP_DISTRIBUTIONLIST_OBJECTCLASS = 'mailList';
LDAP_DISTRIBUTIONLIST_ATTR = 'memberOfGroup';

Lotus Domino#

LDAP_ACCOUNT_OBJECTCLASS = 'dominoPerson';
LDAP_DISTRIBUTIONLIST_OBJECTCLASS = 'dominoGroup';
LDAP_DISTRIBUTIONLIST_ATTR = 'mail';

Zimbra#

LDAP_ACCOUNT_OBJECTCLASS = 'zimbraAccount';
LDAP_DISTRIBUTIONLIST_OBJECTCLASS = 'zimbraDistributionList';
LDAP_DISTRIBUTIONLIST_ATTR = 'zimbraMailForwardingAddress';
LDAP_HELPER_DN = 'uid=zimbra,cn=admins,cn=zimbra';

Note that the GUI grants regular user permissions for everyone authenticated successfully against LDAP. If certain users need auditor rights, then create a group in LDAP, and put the auditors to this group. Then set this value to LDAP_AUDITOR_MEMBER_DN in the customer’s site config:

// the value is case sensitive!
$config['LDAP_AUDITOR_MEMBER_DN'] = 'CN=PilerAuditors,CN=Users,DC=your,DC=domain,DC=com';

After setting the ldap parameters, you have to go to the administration / domain menu on the main admin site, and select the appropriate ldap server for the given domain. If you created a domain before the ldap entry, then simply remove the domain, and recreate it.

A successful login should look like the following in /var/log/mail.log:

Aug  4 06:15:12 676134c1075b piler-webui[425]: ldap query: base dn='ou=users,dc=nodomain', filter='(&(objectClass=inetOrgPerson)(mail=sanyi@aaa.fu))', attr='', 1 hits
Aug  4 06:15:12 676134c1075b piler-webui[425]: ldap auth against '127.0.0.1', dn: 'cn=Sanyi Vagyok,ou=users,dc=nodomain', result: 1
Aug  4 06:15:12 676134c1075b slapd[1357]: <= mdb_equality_candidates: (mail) not indexed
Aug  4 06:15:12 676134c1075b piler-webui[425]: ldap query: base dn='ou=users,dc=nodomain', filter='(|(&(objectClass=inetOrgPerson)(mail=sanyi@aaa.fu))(&(objectClass=zzz)(xxx=sanyi@aaa.fu))(&(objectClass=zzz)(xxx=cn=Sanyi Vagyok,ou=users,dc=nodomain)))', attr='', 1 hits
Aug  4 06:15:12 676134c1075b piler-webui[425]: ldap auth result against 127.0.0.1 / generic_ldap: 1
Aug  4 06:15:12 676134c1075b piler-webui[425]: username=sanyi@aaa.fu, customer=piler, event='logged in', ip=1.2.3.4

A failed login may look like this:

Aug  4 06:30:53 676134c1075b piler-webui[425]: ldap query: base dn='ou=users,dc=nodomain', filter='(&(objectClass=inetOrgPerson)(mail=someuser@aaa.fu))', attr='', 0 hits
Aug  4 06:30:53 676134c1075b piler-webui[425]: ldap auth result against 127.0.0.1 / generic_ldap: 0
Aug  4 06:30:53 676134c1075b piler-webui[425]: username=joska@aaa.fu, customer=piler, event='login failed', ip=1.2.3.4

Authenticating against an IMAP server#

It's also possible to authenticate users against your IMAP server. Set the following in config-site.php:

$config['ENABLE_IMAP_AUTH'] = 1;
$config['IMAP_HOST'] = 'imap.yourdomain.com';
$config['IMAP_PORT'] = 993;
$config['IMAP_SSL'] = true;

Authenticating against Google Workspace (formerly GSuite)#

https://mailpiler.com/authenticate-against-google-workspace-formerly-gsuite/

Use Azure AD for single sign-on (SSO)#

https://mailpiler.com/using-azure-ad-single-sign-on-sso/

Use AWS Cognito for single sign-on (SSO)#

https://mailpiler.com/using-aws-cognito-single-sign-on-sso/

Customizing the GUI#

Using the administration / customers menu the GUI allows you to override some logos, branding URL, and optionally the text and background colours in the menu bar.

To override the logo on the login screen for a given customer, set the following in /etc/piler/sites/config-site-customerid.php eg.:

/etc/piler/sites/config-site-fictiveco.php:

$config['SITE_LOGO_LG'] = '/path/to/fictiveco.jpg';

To override the logo in the menu bar, create a transparent image with up to 125px wide and up to 25px tall, then click on select in the customer settings menu. You may also define the branding URL assigned to the logo, and override the background and text colour in the menu bar.

The default values can be found in /var/piler/www/config.php:

$config['BRANDING_TEXT_COLOUR'] = '#FFFFFF';
$config['BRANDING_BACKGROUND_COLOUR'] = '#2B5CBE';
$config['BRANDING_LOGO'] = '/view/theme/default/assets/images/archive-logo-sm.png';
$config['BRANDING_FAVICON'] = '/view/theme/default/assets/ico/favicon.png';
$config['BRANDING_URL'] = ''; // the URL when hovering over the logo
$config['SUPPORT_LINK'] = ''; // an additional link in the user menu

You may override any of these default values by putting the appropriate definitions to /etc/piler/config-site.php, eg:

$config['BRANDING_BACKGROUND_COLOUR'] = '#B8377C';
$config['BRANDING_URL'] = 'http://www.mycompany.com/';
$config['SUPPORT_LINK'] = 'http://support.mycompany.com/';

Note that it’s possible to specify a custom css for the given customer by adding the following to override any default css property:

$config[‘CUSTOM_CSS’] = ‘/path/to/fictiveco.css’;

How to increase the number of search hits:

$config[‘MAX_SEARCH_HITS’] = 20000;

Increase the number of hits per page:

$paging = [10,20,30,50,100];
$config['PAGE_LEN'] = 100; // the default for users

Then let the users override the hits per page (=page length) value in the user settings menu.

Administrator tasks#

Users, groups, and rights#

Piler allows you to access only your emails you either sent or received. It's often required to grant permission to read emails of other users. Let's say Alice wants to see Bob's and Jim's emails.

To do so create a group, type its name, eg. group1, then enter Alice's email address to the "Email addresses**" textarea. Finally type Bob's and Jim's email addresses to the "Assigned email addresses**", and click on Add or Modify.

Thus when Alice logs in both Bob's and Jim's email addresses are added to her allowed to see email addresses, so Alice can read Bob's emails, and Jim's emails, and of course her own emails as well.

Administrator permissions#

The administrator account (eg. admin@local) is used to administer piler only. It's not a super powerful account to see anyone's emails that's why it can't see the Search menu at all.

If you need a user who can access anyone's emails then grant him AUDITOR privileges on the user settings page. Note that by default there's no such an account, you need to create it.

The administrator role may see system statistics, accounting summary, edit user / group settings, policies, see audit logs, etc.

Accounting#

Piler keeps track of how many emails it has archived so far, and it’s able to show this information grouped by email addresses or domains. Note that Piler creates accounting data only for the domains configured in the administration / domain menu. The accounting runs once a day.

Audit logs#

The GUI keeps track of what users do and when they do. Every user's action involves an audit record that the gui stores into the audit SQL table. Thus there's an audit trail of every user activity, eg. searching for some messages, viewing a message, downloading another one, etc. Also the gui logs when a user logs in or logs out.

The following information is logged:

Administrators and auditors are able to search within the audit logs, and even export the audit trail as a CSV file.

The search field is very similar to the quick search for emails, and the parser tries to figure out what you look for. If it has a format of a date (eg. YYYY.MM.DD or YYYY-MM-DD) then it's a date, if it has an @ symbol then it's an email, dotted quads are IP-addresses, and search|view|download|... are actions.

You may use wildcards by using the asterisk (*) character. Eg. 2018.08.* searches for the whole August of 2018. jack@yourdom* Searches for user jack in domains starting with “yourdom”. If you want to specify only the user part, then you still must include the @-symbol otherwise the parser won't recognize it as an email address: jack*@. IP-addresses can be “wildcarded” right after a dot, eg. 1.2.3.* or 1.2.*.*. Note that 123.123.12*.* is not a valid expression.

A slightly complex audit search to find the login, view and download attempts for the local auditor in this month from 31.0.0.0/8 address space:

audit*@ 2018.08.* login view download 31.*

If you don't want or need auditing, then have the provider to turn it off for you:

$config['ENABLE_AUDIT'] = 0;

Securing the archive#

Securing the GUI#

Enable hCAPTCHA#

This feature requires piler enterprise version 1.7.3+.

Note that this feature obsoletes the previous secureimage based captcha.

Register a plan at https://www.hcaptcha.com/. There's a free plan up to 1 million requests per month.

Create a site key and a secret, as well as add each hostname you want to protect.

Finally set the site key and secret in /etc/piler/config-site.php:

$config['HCAPTCHA_SITEKEY'] = 'aaaaa-bbbbb-......';
$config['HCAPTCHA_SECRET'] = '0x...........';

Services providers using a multitenant license may be selective to enable this feature for specific tenants as well by putting these variables to the tenant's site specific config.

Securing the piler-smtp daemon#

Rules and policies#

Piler supports two kinds of policies or rulesets. The exclusion rules specify what NOT to archive. The retention rules specify how long to keep messages in the archive.

Customer specific rules are stored in the customer’s database in some mysql tables on the GUI node. The worker nodes synchronize these tables every hour using a piler cron job.

Warning! Currently only the provider admins can set or fix both archiving and retention rules, because a single miswritten rule may lead to loss of important emails.

When piler starts it reads all the rules and compiles them at startup. If the piler daemon gets a HUP signal, then it re-reads the rules.

Exclusion rules#

You may define rules to prevent piler and pilerimport to archive certain messages, eg. having no subject, too big, recognized spam, etc.

You may specify the From:, To/Cc:, Subject:, message size, attachment type and size value. Piler uses the TRE regex library to test whether the given rule matches the processed message. If so, then neither piler nor pilerimport will archive it, but rather discard it after logging.

Some examples:

Pattern Matches on
subject: ^ {0,}$ messages with no subject
size: > 1000000, attachment type: video/ messages bigger than 1 MB + having a video attachment
size: < 500 too short spam probes, not having a body
subject: [SPAM] recognised spam emails
BUY( NOW){0,1} (viagra|cialis) “buy” or “buy now” viagra/cialis spam

You are encouraged to test your rules whether they work as expected. Use the pilertest utility to test a single EML format message against your exclusion rules. Pilertest prints the first matching exclusion rule (if there’s any), eg. the following mail is caught by a rule matching emails having no subject:

pilertest -W customerid -m message.eml

locale: hu_HU
message-id: <586387.60094.qm@web83907.mail.sp1.yahoo.com>
from: *Queen Oneil queenbaby3@att.net queenbaby3 att net (att.net )*
to: *undisclosed recipients ()*
reference: **
subject: **
sent: 1293679136
hdr len: 3007
digest: 0230af406d0e03e56ed63eeb8e0891ed6a97f49d297f42c0d2565cb770311323
rules check: from=,to=,subject=^$,size0,att.type=,att.size0
attachments:
direction: 0
spam: 0

Retention rules#

How long shall piler retain your messages? You may control it with the retention rules. The format is exactly the same as with the exclusion rules: you can define the From:, To/Cc:, Subject:, message size, etc. and the retention period in days.

Every time piler archives a message, it assigns a retention period according to the retention rules or applying the default value (2257 days = 7 years + 2 days, unless you change it in piler.conf). After the retention period has expired the piler utilities will remove the message from the archive.

It's recommended to create your retention policy before deploying any archiving solution.

Note that the retention period is included in the per message verification digest, so the retention period should not be changed after the message has been archived.

Piler also compiles a list from the retention rules, so whenever you change them, be sure to click on the “Apply settings” red button to send a HUP signal to piler daemon.

Note that if you want to keep your emails forever for the given customer, then be sure to disable purging on the admin panel!

What to do with spam#

It's a waste to archive junk messages, so the best solution is to prevent them from entering the archive. This can be achieved by redirecting spam messages to a quarantine, so only good emails can get to the archive.

If this setup is not feasible, you may create an exclusion rule that keeps spam emails out of the archive. Or you may create a retention rule that no spam is to be retained beyond 30 days. Unfortunately these may be error prone, think about a false positive error.

Backup and restore#

Backup is an essential part of the business continuity. Make sure that you backup the following directories and data to ensure that you will be able to restore in case of a problem.

The /etc/piler directory contains piler configuration, key file, certificate, etc. Make sure that you never lose /etc/piler/piler.key, otherwise you won’t be able to decrypt the archived emails!

The /var/piler/store directory stores the archived emails. An email is splitted to several parts. A filename ending with .m stores the message itself without the attachments. The attachments are stored in /var/piler/astore directory in files named as -, eg. d440749d9a64d44e1fd81e63a5e3d1cfa57edd8884db3013d22d831fc21228cc-000050930.

The stored .m files are organized to sub directories, eg.

/var/piler/store/<server_id>/<customer_id>/58f
                                          /590
                                          /591

Notice the subdirectories consisting of three hexadecimal numbers (58f, 590 and 591 in the example above). They hold ~12 days of data. After ~12 days (give or take) a new directory is created, and piler starts to put new emails to the new directory, and won’t touch any of the older directories ever. This behaviour allows you to create an incremental backup procedure.

The index data follows a main - dailydelta - delta indexing scheme. The main files change once a day, the dailydelta files change every 30 mins. You don’t have to backup the delta files, it’s enough to backup the main files (ie. /var/piler/manticore/customerid/main1.*).

Note that the index data can be regenerated, however the more emails you have, the longer the process takes, so it’s worth to backup the manticore files as well.

Finally you should backup the mysql data. Piler stores crucial metadata, users, rules, audit data, etc. in a mysql database required to operate properly. Make sure to never lose the mysql databases!

You may choose to backup the emails as well. The pilerexport utility allows you to export yesterday’s emails to the current directory, eg.

cd /path/to/backup/somecustomer
pilerexport -W somecustomer -a 2017.04.17 -b 2017.04.17.

The above command dumps all emails for ‘somecustomer’ to /path/to/backup/somecustomer archived on 2017.04.17. The filenames consist of the message serial id + .eml suffix, eg. 123.eml.

Important! DON’T export emails of different customers to the same target directory because the dumped eml files will be overwritten! Each customer should have its own dump directory.

The restore procedure simply involves importing these emails.

You may even set up a secondary piler installation, see the next chapter.

Remove a customer’s data#

When you remove a customer on the gui node, then the customer’s reference is removed only, and all their emails, manticore index and sql database remain intact. To remove the customer’s data, follow the steps below on how to remove a customer called fictive.

On all nodes including both gui and worker nodes (Note that steps 5-6 may take a long time):

stop searchd
rm -rf /var/piler/manticore/fictive
start searchd
mysql>drop database fictive;
rm -rf /var/piler/astore/00/fictive
rm -rf /var/piler/store/00/fictive

On the GUI node only:

Setup a secondary piler site#

There’s another option to mitigate the effects of a site disaster. Let’s say you have a Piler installation in Azure with a gui node, an smtp gateway, and 5 worker nodes.

Now create another site at AWS (or your favourite provider. It can even be your datacenter if you will), and perform the same installation of the gui node, the smtp gateway, and the worker nodes. Make sure that you use the same internal IP addresses for both sites (ie. Azure and AWS in this example) for the worker nodes.

Finally you have to configure the smtp gateway on each site to forward a copy of emails to the other site thus replicating emails automatically to the other site. When the azure site goes down or is destroyed, then you simply redirect customers to the backup site by fixing the DNS entries.

One things is not covered in the documentation is that you have to keep the configurations of each sites synchronized, ie. set the same exclusion rules, same retention periods, etc. The current version of Piler doesn’t support this automatically.

Best practices#

A few recommended steps on how to operate piler.

Set a reasonable default retention time in piler.conf that might be valid to most of your customers.

The piler.conf files should be identical on all worker nodes.

The piler.key file should be the same on all worker nodes.

Use DNS domain names instead of IP-addresses.

Monitor the piler nodes, including the gui node(s), the workers, and the smtp gateway as well. The monitoring should include checking port 25, 80 (or 443), 9312 availability, enough free disk space, whether mysql server is running, cpu, memory and disk io, etc.

Never ever run out of free disk space. Not even under /tmp. Note that manticore requires some additional temporary disk space while indexing besides the index files.

Implement a central syslog server or other log collecting solution. All hosts should send their mail related logs to a central log processing server.

Backup all piler related configuration as well as data. This includes the following directories:

/etc/piler
/var/piler/astore/<server_id>
/var/piler/store/<server_id>
/var/piler/manticore
/var/piler/sphinx # for older deployments using sphinx

And backup the mysql databases as well!

It’s recommended to put the worker nodes (or even all piler hosts) behind a firewall. See the Security considerations chapter in the Installation guide.

The smtp gateway should support STARTTLS to secure emails on the wire.

Buy a wildcard certificate for your domain, and you may use it to secure both GUI web access and smtp gateway. You may even use it on the worker nodes to secure smtp. In this case put both the private key and the certificate to /etc/piler/piler.pem, then restart piler-smtp process.

In case of an issue, always check the logs. When piler encounters a problem, it syslogs the issue, it gives a clue what went wrong. Check the nginx logs as well.

Also in case of a problem run the health check utility: /usr/libexec/piler/health-check.sh. It checks a few settings the Piler support may be interested in.