Archiving

Description

When we talk about “Archiving” in Cyrus (from 3.0 onwards) we’re not talking about data retention, what we’re really talking about is time-tiered storage. Elsewhere in these documents we discuss storage tiering based on things like space limitations or scalability. Archiving is somewhat like this, but with an accompanying configuration syntax which determines the management of the tiered data on an ongoing basis.

Purpose

The general intent of archiving, as deployed within Cyrus, is to allow administrators to dictate that there be two tiers of data storage for each mail spool partition: “current” and “archive.” The common use case is to use higher speed storage media for the former, and lower cost media for the latter; Current goes on SSDs, and Archive goes on traditional spinning media.

Enabling

Archive operation in Cyrus is enabled via the archive_enabled setting in imapd.conf(5):

archive_enabled: 0

Is archiving enabled for this server. You also need to have an archivepartition for the mailbox. Archiving allows older email to be stored on slower, cheaper disks - even within the same mailbox, as distinct from partitions.

Defining

The configuration suite provides directives to define these partitions:

archivepartition-name: <none>

The pathname of the archive partition name, corresponding to spool partition partition-name. For any mailbox residing in a directory on partition-name, the archived messages will be stored in a corresponding directory on archivepartition-name. Note that not every partition-name option is strictly required to have a corresponding archivepartition-name option, but that without one there’s no benefit to enabling archiving.

Controlling

And to control the criteria used to manage migration of data between partitions:

archive_after: 7d

The duration after which to move messages to the archive partition if archiving is enabled.

For backward compatibility, if no unit is specified, days is assumed.

archive_maxsize: 1024

The size in kilobytes of the largest message that won’t be archived immediately. Default is 1Mb

archive_keepflagged: 0

If set, messages with the \Flagged system flag won’t be archived, provided they are smaller than archive_maxsize.

Note

Further explanation is probably required for the archive_maxsize option. The value is a threshold. Messages larger than this threshold will be immediately put onto the archive partition, rather than waiting archive_days number of days. This is to keep the high speed storage available for the largest number of “current” messages.

So archive_maxsize is not the maximum size of messages which will be archived, but rather is the threshold above which they will immediately be.

Performing

Finally, the actual migration is handled by invoking cyr_expire(8) with its -A flag, as shown here in a sample snippet from cyrus.conf(5) (but could also be done via cron):

EVENTS {
    ...
    archive      cmd="cyr_expire -A 7d" at=0403
    ...
}