Skip to content

Using S3 object store#

By default piler stores email files on the local filesystem under /var/piler subdirectories, thus requiring lots of disk space.

Piler is able to store files on S3 compatible object stores, eg. Amazon S3, Wasabi, Exoscale, etc. Or you can even set up your own S3 object store with 3rd party software like Minio. Note that you should decide where to store the email files before starting archiving. Piler doesn’t support a mixed mode where some emails are stored locally, and some emails on an S3 object store.

When using S3 storage the piler daemon writes all files to /var/piler/s3 directory. Then a piler service processes these files, and upload them to the S3 store.

Piler will upload all files for a given tenant to a single bucket only.

Enable the S3 storage#

To enable using an S3 store, set the following in /var/piler/.env. piler, piler-ui, the piler-s3 service (s3uploader), purge, reindex and pilerexport all read their settings from this file:

S3_ENDPOINT=your-s3-host.domain.com:9000
S3_ACCESS_KEY=youraccesskey
S3_SECRET_KEY=yoursecretkey
S3_USE_SSL=true
S3_REGION=us-east-1
S3_WORKERS=10
S3_BUCKET_PREFIX=
RETRIEVER_METHOD=s3

RETRIEVER_METHOD=s3 makes piler-ui read the emails from the S3 store. S3_WORKERS is the number of parallel uploads of s3uploader.

S3_ENDPOINT should be the S3 hostname and port. Eg. Minio uses port 9000. For Wasabi, it might be s3.us-west-1.wasabisys.com.

Set S3_USE_SSL=false if your S3 host does NOT support TLS.

By default S3_BUCKET_PREFIX is empty. If you need to specify one, then make sure it conforms to domain names.

Notes: you must NOT use the dash (-) character, however you may use a trailing dot, eg.

S3_BUCKET_PREFIX=someprefix.

The bucket of a tenant is S3_BUCKET_PREFIX + tenant id: s3uploader uploads to that bucket, and piler-ui, purge, reindex and pilerexport read from it. S3_PREFIX (a key prefix inside the bucket) is not supported, s3uploader refuses to start with it.

Buckets and permissions#

Piler doesn't create buckets. Create the bucket of a tenant (S3_BUCKET_PREFIX + tenant id) before the tenant receives email, with the policy, encryption, versioning or object lock you need. Backups of the nodes don't include the buckets, see S3 object storage.

s3uploader doesn't check the bucket before uploading; the upload of a file to a missing bucket fails, and the file goes to the error directory (/var/piler/error) at once. The log names the bucket ("the bucket does not exist, create it"); access denied and invalid keys are logged the same way. Once the problem is fixed, move the files back to be uploaded:

mv /var/piler/error/<tenant id>-* /var/piler/s3/0/

Until a file is uploaded, its email is found by the search, but it can't be opened: piler-ui reads the emails from S3 only. A file normally stays in /var/piler/s3 for seconds (S3_SCAN_INTERVAL, 10 seconds by default), a few minutes at most while s3uploader retries a network or S3 server error. Older files mean that s3uploader isn't running or can't upload.

The S3 permissions the piler tools need on the tenant buckets:

Permission Used by
s3:PutObject s3uploader
s3:GetObject piler-ui, pilerexport, reindex; s3uploader with S3_SKIP_EXISTING_OBJECTS or S3_VERIFY_AFTER_UPLOAD
s3:DeleteObject purge
s3:ListBucket purge -orphans, the S3 statistics; optional for S3_SKIP_EXISTING_OBJECTS (without it a missing object is a 403, and the file is uploaded)

Before 2.3.0 the piler-s3 service read /var/piler/.env-s3. If you upgrade from an earlier version, see S3 storage settings.

S3 settings for oracle cloud:

S3_ENDPOINT=frnsjgsfvvgl.compat.objectstorage.eu-frankfurt-1.oraclecloud.com
S3_REGION=eu-frankfurt-1

S3 settings for Cloudflare bucket using EU jurisdiction:

S3_ENDPOINT=zzzzzzzzzzzzzzzzz.eu.r2.cloudflarestorage.com
S3_REGION=auto

S3 settings for Backblaze bucket using EU-central:

(Note: Backblaze doesn't accept dot (.) in the bucket name!)

S3_ENDPOINT=s3.eu-central-003.backblazeb2.com
S3_REGION=us-east-1

S3 settings for Hetzner object store at fsn1 location:

S3_ENDPOINT=<your hetzner id>.fsn1.your-objectstorage.com
S3_REGION=us-east-1

Enable the the piler-s3 service#

cd /etc/systemd/system
ln -sf /usr/libexec/piler/piler-s3.service .
systemctl daemon-reload
systemctl enable piler-s3
systemctl start piler-s3

S3 statistics#

To allow S3 statistics you need to install the minio client:

curl -o /usr/local/bin/mc https://dl.min.io/client/mc/release/linux-amd64/mc
chmod +x /usr/local/bin/mc

Then create a config file for minio:

su - piler
mkdir -p /var/piler/.mc/

Create /var/piler/.mc/config.json. In the below example we used the following parameters (be sure to use your own settings):

S3_ENDPOINT=https://zzzzzzzzzzzzzzzzz.eu.r2.cloudflarestorage.com
S3_ACCESS_KEY=aabbaabbaabb
S3_SECRET_KEY=deadbeef0123456789abcdef

/var/piler/.mc/config.json:

{"version":"10", "aliases": {"minio": {"url":"https://zzzzzzzzzzzzzzzzz.eu.r2.cloudflarestorage.com","accessKey":"aabbaabbaabb", "secretKey": "deadbeef0123456789abcdef", "api": "S3v4", "path": "auto"}}}

Then finally add the following to piler's crontab entries:

30 * * * * /usr/libexec/piler/s3-bucket-stat.sh minio

This will update the s3_bucket_stats table in the mysql piler database every hour. The result looks like the following. These details will be displayed at the tenant listing.

MariaDB [piler]> select * from s3_bucket_stat;
+---------+----------+---------+---------------------+
| bucket  | size     | objects | t                   |
+---------+----------+---------+---------------------+
| fictive | 20481360 |    2978 | 2024-02-06 10:30:08 |
+---------+----------+---------+---------------------+
1 row in set (0.001 sec)