v0.6.1 IMAP Thunderbird and K-9 mail observations

[cbrake]

Behavior in Thunderbird seems much better now. Before, messages would not get marked as read, and when I would delete messages, some would come back. I don't see any problems in Thunderbird with 0.6.1 -- nice work on the IMAP improvements.

However, I am seeing some problems in the Android K-9 client where it does not always fetch new messages in the inbox. See comparison of T-bird and K-9:

I've tried stopping/restarting K-9, but it simply seems to refuse to fetch new inbox messages. K-9 worked pretty will with maddy 0.5.4. Is anyone else using Android K-9 with Maddy?

Overall, it seems like a big step forward with IMAP performance.

My config:

## Maddy Mail Server - default configuration file (2021-08-16)
# Suitable for small-scale deployments. Uses its own format for local users DB,
# should be managed via maddyctl utility.
#
# See tutorials at https://maddy.email for guidance on typical
# configuration changes.
#
# See manual pages (also available at https://maddy.email) for reference
# documentation.

# ----------------------------------------------------------------------------
# Base variables

$(hostname) = <deleted>
$(primary_domain) = <deleted>
$(local_domains) = $(primary_domain) <deleted>

tls file <deleted>

# ----------------------------------------------------------------------------
# Local storage & authentication

# pass_table provides local hashed passwords storage for authentication of
# users. It can be configured to use any "table" module, in default
# configuration a table in SQLite DB is used.
# Table can be replaced to use e.g. a file for passwords. Or pass_table module
# can be replaced altogether to use some external source of credentials (e.g.
# PAM, /etc/shadow file).
#
# If table module supports it (sql_table does) - credentials can be managed
# using 'maddyctl creds' command.

auth.pass_table local_authdb {
    table sql_table {
        driver sqlite3
        dsn credentials.db
        table_name passwords
    }
}

# imapsql module stores all indexes and metadata necessary for IMAP using a
# relational database. It is used by IMAP endpoint for mailbox access and
# also by SMTP & Submission endpoints for delivery of local messages.
#
# IMAP accounts, mailboxes and all message metadata can be inspected using
# imap-* subcommands of maddyctl utility.

storage.imapsql local_mailboxes {
    driver sqlite3
    dsn imapsql.db
}

# ----------------------------------------------------------------------------
# SMTP endpoints + message routing

hostname $(hostname)

table.chain local_rewrites {
    optional_step regexp "(.+)\+(.+)@(.+)" "$1@$3"
    optional_step static {
        entry postmaster postmaster@$(primary_domain)
    }
    optional_step file /etc/maddy/aliases
}

msgpipeline local_routing {
    # Insert handling for special-purpose local domains here.
    # e.g.
    # destination lists.example.org {
    #     deliver_to lmtp tcp://127.0.0.1:8024
    # }

    destination postmaster $(local_domains) {
        modify {
            replace_rcpt &local_rewrites
        }

        deliver_to &local_mailboxes
    }

    default_destination {
        reject 550 5.1.1 "User doesn't exist"
    }
}

smtp tcp://0.0.0.0:25 {
    limits {
        # Up to 20 msgs/sec across max. 10 SMTP connections.
        all rate 20 1s
        all concurrency 10
    }

    dmarc yes
    check {
      require_mx_record
      dkim
      spf {
        debug no
        enforce_early no
        none_action ignore
        neutral_action ignore
        fail_action quarantine
        softfail_action ignore
        permerr_action quarantine
        temperr_action quarantine
      }
    }

    source $(local_domains) {
        reject 501 5.1.8 "Use Submission for outgoing SMTP"
    }
    default_source {
        destination postmaster $(local_domains) {
            deliver_to &local_routing
        }
        default_destination {
            reject 550 5.1.1 "User doesn't exist"
        }
    }
}

submission tls://0.0.0.0:465 tcp://0.0.0.0:587 {
    limits {
        # Up to 50 msgs/sec across any amount of SMTP connections.
        all rate 50 1s
    }

    auth &local_authdb

    source $(local_domains) {
        check {
            authorize_sender {
                prepare_email &local_rewrites
                user_to_email identity
            }
        }

        destination postmaster $(local_domains) {
            deliver_to &local_routing
        }
        default_destination {
            modify {
                dkim $(primary_domain) $(local_domains) default
            }
            deliver_to &remote_queue
        }
    }
    default_source {
        reject 501 5.1.8 "Non-local sender domain"
    }
}

target.remote outbound_delivery {
    limits {
        # Up to 20 msgs/sec across max. 10 SMTP connections
        # for each recipient domain.
        destination rate 20 1s
        destination concurrency 10
    }
    mx_auth {
        dane
        mtasts {
            cache fs
            fs_dir mtasts_cache/
        }
        local_policy {
            min_tls_level none
            min_mx_level none
        }
    }
}

target.queue remote_queue {
    target &outbound_delivery

    autogenerated_msg_domain $(primary_domain)
    bounce {
        destination postmaster $(local_domains) {
            deliver_to &local_routing
        }
        default_destination {
            reject 550 5.0.0 "Refusing to send DSNs to non-local addresses"
        }
    }
}

# ----------------------------------------------------------------------------
# IMAP endpoints

imap tls://0.0.0.0:993 tcp://0.0.0.0:143 {
    auth &local_authdb
    storage &local_mailboxes
}

[cbrake]

Since I posted the above, I'm now seeing problems in Thunderbird:

• not all messages show up in inbox
• filtered messages in thunderbird don't show up in folders they were filtered to

Still digging ...

[cbrake]

In TB, occasionally when I move a message to a folder, it does not show up in the folder.

Also, I used to have a newsletter folder with 100's of messages in it, and it now shows three.

Once #509 is resolved, will see if what I'm seeing in email client matches server.

I realize IMAP is beta, so not complaining -- just reporting observations.

[foxcpp]

I tried running K-9 and Thunderbird with a clean setup and I do not see any significant issues when adding, moving, deleting messages.

A protocol trace would help to identify exactly how maddy misbehaves in your setup.
Configuration changes:

storage.imapsql local_mailboxes {
  debug
  ...
}
imap tls://0.0.0.0:993 tcp://0.0.0.0:143 {
  io_debug
  debug
  ...
}

Then upload logs somewhere. They may include bits of message data, feel free to search and replace any usernames, passwords, etc you do not want exposed and/or you can email them to [email protected] if you do not want to post it publicly.

[cbrake]

Thank you for the suggestions.

A few more observations:

• I have 4 users on this mail server, and all of them saw similar things -- messages disappearing from the inbox
• my son who uses apple mail readers on iPhone and Macbook also had problems with messages not showing up in his inbox. In one experiment, I sent him a new message, and then the entire inbox showed up again.
• I had a newsletters folder that had lots of messages in it, most of them have disappeared and now there are only 3.

I used the maddy command line to list messages (thanks for the fix for that) in the Newsletters folder and go 97 messages:

maddy-git imap-msgs list [email protected] Newsletters | grep UID | wc -l
97

However, in TB on multiple machines I see the following:

On phone (K-9), I did not have this folder downloaded, and when I clicked on it, it seemed to load the entire folder. So it seems the mail is still on the server, but not synchronized correctly to the IMAP clients.

I decided to switch to https://nixos-mailserver.readthedocs.io/en/latest/ for now, but will leave the old server running to test future versions. Will also try to get some logs when I can find some time.

[cbrake]

#512 fixes the IMAP issues I was seeing here.