Large deployments: sizing#
This documentation applies to Piler enterprise edition 2.2.4+
Revision #1
Publication date: 2026-OCT-18
How to estimate the storage and the number of nodes of a large deployment. Your mail volume, email size, retention and user activity set the numbers; a proof of concept with a sample of your email traffic confirms them.
Start from the number of emails#
Everything grows with the number of emails archived:
emails per day = users × emails per user per day
raw volume = emails × average email size
A busy corporate user receives about 100 emails a day. Where many mailboxes are less active, as is typical for hosted email, the average is lower, e.g. 30. Count incoming and outgoing email, and the average size (often 50-150 kB).
Storage per million emails#
| Per million emails | Notes | |
|---|---|---|
| Stored emails | 30-40% of the raw volume | After deduplication (one copy of an email sent to several recipients of the archive, one copy of an identical attachment) and compression; 27-41% in existing installations |
| Search index (Manticore) | ~1 GB | |
| Database (MySQL / MariaDB) | ~1.8 GB |
The index and database figures were measured on an installation of 44 million emails. For example, 1 million emails of 100 kB are ~100 GB raw, 30-40 GB stored, plus ~2.8 GB of index and database.
Multiply by the emails per year and the retention period in years, and by two if a DR site stores everything as well.
Incoming rate: the average is emails per day / 86,400 per second; plan for
peaks of 3-5 times the average.
What this means#
- A proof of concept sets the number of worker nodes. Above a few hundred million emails a year, the number of workers and their resources should be confirmed with a sample of your email traffic.
- Retention drives the storage as much as the volume. A few years of retention at a high volume means hundreds of terabytes of stored emails, and billions of database rows across the workers.
- Fast storage for the databases. MySQL and Manticore on each worker need SSD volumes; the stored emails can live on cheaper volumes or on object storage. All of them on replicated storage, see protecting the workers.
Semantic search storage#
Semantic search is optional and disabled by default. Full-text search covers every archived email without it. We recommend enabling semantic search only after its storage and compute have been planned for, as they are substantial at a high volume, and per tenant, for the tenants that will make use of it.
Semantic search (part of the Scanner add-on) stores an embedding for every email it covers, in a separate Manticore table with a vector index. This comes on top of the full-text index and the database above.
Each embedding takes dimensions × 4 bytes, plus about 10-20% for the
vector index; the sender and recipient field stored with it adds little. The
dimensions are set by the embedding model you choose:
| Model (examples) | Dimensions | Per email | Per 1 million emails |
|---|---|---|---|
multilingual-e5-small, all-minilm |
384 | ~1.5 KB | ~1.7-1.9 GB |
multilingual-e5-base, nomic-embed-text |
768 | ~3 KB | ~3.4-3.7 GB |
bge-m3, multilingual-e5-large, mxbai-embed-large |
1024 | ~4 KB | ~4.5-4.9 GB |
| larger API models | 1536 | ~6 KB | ~6.8-7.4 GB |
For a non-English archive choose a multilingual model:
multilingual-e5-small (384) is the compact choice, multilingual-e5-base
(768) or bge-m3 (1024) give better quality at two to three times the
storage. all-minilm, nomic-embed-text and mxbai-embed-large are best
for English. See semantic search.
- Plan memory as well: the vector index serves searches fastest when it fits in memory.
- The storage can be limited: semantic search can be enabled per tenant, and only for the tenants that need it.
- Changing the model later means recreating the table and computing the embeddings of the archived emails again: choose the model before going live.
UI nodes#
The number of UI nodes depends on how your users search: occasional look-ups by a few users need far less than thousands of users searching all day. Two UI nodes give redundancy and serve light use; more can be added at any time as the load grows.
Resources per node#
The minimum for a Piler node is 2 vCPUs, 2 GB of memory and 200 GB of disk, see the overview. At a large scale the workers need far more, mostly for the databases: memory for the search index and the MySQL buffer pool, and fast storage.
To estimate the storage for your own volume, see the email archive sizing calculator.