Skip to content

Queue handling in ZoneMTA

Jump to bottom
Andris Reinman edited this page Oct 30, 2016 · 4 revisions

ZoneMTA uses LevelDB to store and manage message queue on disk. Mail queue has a few requirements which makes it hard to use a random database backend to store the queue.

  • Write heavy. Messages can be queued in large amounts, especially if you are sending to lists
  • Light reads. At best you only need to read message data once
  • Quick deletes. In a way the message queue acts more like a cache. You write to it, read once and immediately delete. The average lifetime of a value might be just a single second.
  • Variable sizes. A queued message might be anything from few hundred bytes to hundred megabytes in size. There is no average message size, so it is hard to optimise for it

ZoneMTA stores message body in chunks in the receiving order where every chunk gets indexed by an incrementing ID value

  1. message [MSGID] 000001 (64kb chunk)
  2. message [MSGID] 000002 (64kb chunk)
  3. message ...
  4. message [MSGID] nnnnnn (remaining chunk)

Once all body chunks are successfully stored, ZoneMTA adds two additional keys to LevelDB for the message

  1. message [MSGID] # (indicates that the message exists, stores creation time)
  2. message [MSGID] * (JSON serialised metadata about the message, including parsed message headers)

Next, ZoneMTA stores recipient data where every recipient has its own key. Recipient keys are ordered by MSGID which in itself is an incrementing value, so messages inserted earlier are ordered before messages added later.

  1. delivery-zone [ZONE] [MSGID] [email protected]

When ZoneMTA is looking for new messages to deliver, it iterates through recipient list by Zone name

"delivery-zone [ZONE] " ... "delivery-zone [ZONE] ~"

This iteration starts with recipients for the oldest message and continues up to the latest message.

If a recipient is fetched from the pending queue, ZoneMTA also fetches message metadata from the metadata key. Next the MX for the recipient is resolved and a SMTP connection is initiated. Once SMTP transaction is set up, ZoneMTA fetches all message chunks in the storing order, passes these through the SMTP encoder (ensures \r\n line breaks, escapes dots in the beginning of the lines, adds finishing dot) and pipes it to the receiving SMTP server.

Once a message is delivered ZoneMTA checks if there are any recipients remained for the message and if not, then deletes all message related keys from the DB.

Clone this wiki locally