The future of Himalaya

[soywod]

So far, I am happy with the direction Himalaya is taking. People seem to be interested in the concept of a CLI email client. I hope to reach the v1.0.0 soon. But I see 2 big issues that raise questions about the future of Himalaya:

• Everything is quite slow. This is because Himalaya does a request for every single action (list emails, read email, fetch mailboxes etc). We could reduce delays by bringing the emails locally (using an IMAP synchronizer like isync or OfflineIMAP), but I see 2 drawbacks: the UX (users need to configure Himalaya AND one other tool in order to have an operational environment) and the real-time (both tools sync emails at regular intervals, usually with a cron). The other solution is to introduce some kind of cache inside Himalaya. Which means that Himalaya becomes also an IMAP synchronizer.
• The contacts completion. This implies to communicate with a CardDAV server. I guess we also need to synchronize them locally to avoid delays. The only tool I know doing so is vdirsyncer, but it is written in Python (I would prefer a more solid language) and it is also sth to configure outside of Himalaya. So it means to implement a carddav synchronizer and a carddav CLI (to query the database).

There is also a third (optional) point: the events. It could be nice to have a synchronizer + a CLI to manipulate a calendar (CalDAV). Like for the CardDAV, the only lib I know doing so is khard, which is also written in Python.

---

My conclusion: there is a real lack of tools in this domain. They all depend on each other, and they all serve the same purpose: to manage personal informations. So why not turning Himalaya into a suite of PIMs (Personal Information Managers)?

[srid]

I haven't experienced the slowness because in my case (and that of every ProtonMail users) IMAP connections are all local.

The protonmail bridge already acts as such a local synchronizer. Himalaya connects to it via 127.0.0.1. If himalaya implements a cache, I would like a way to disable it permanently in the config. I personally like the simple core of himalaya: it does one thing only, and that's managing IMAP email.

So why not turning Himalaya into a suite of PIMs

I'd be happy as long as I can continue to use himalaya for email only, sans contacts or calendar. The beautify of himalaya is that it abstracts out email, so I can build further tools (be it scripts or whatever) on top it. If you choose to go the PIM route, I hope the email aspect is at least available as a different sub-program entry point? Perhaps himalaya-imap (with the larger himalaya executable acting as a traditional email program).

In regards to PIM, that is generally a good idea (email is eternal and open; it is good to use it for our personal information, rather than some black box proprietary system). I personally would be most interested in a task manager, GTD style. cf. those people doing GTD on top of Outlook. But of course this as can well be implemented on top of (or, as an extension of) himalaya.

[soywod]

Thanks for your feedback!

I haven't experienced the slowness because in my case (and that of every ProtonMail users) IMAP connections are all local.

Indeed, it is because it acts like a synchronizer already. So ProtonMail users are not concerned by this feature.

If himalaya implements a cache, I would like a way to disable it permanently in the config

I do not plan to force users to use this or this feature. I see them more like flags to enable/disable.

If you choose to go the PIM route, I hope the email aspect is at least available as a different sub-program entry point?

Yes, I planned to use the Cargo's workspace feature, so it should build one executable by crate.

I personally would be most interested in a task manager

In fact, I forgot to mention it in my initial message (I will update it). I already have a task & time manager I use in production: https://github.com/soywod/unfog, and I was thinking to import it (and maybe to rewrite in Rust, so I can use it as a lib rather than communicating in JSON between both).

[soywod]

@srid I was able to set up a nice local environment with isync and dovecot. I found a nice trick on a blog post: instead of synchronizing every X mins your emails with a cron, you just need to synchronize them whenever a change occurs on the server (it requires the server to implement the IDLE mode, which is sth very common nowadays). Himalaya has already this notify feature based on the IDLE mode. I introduced a new setting called idle-hook-cmds that takes a list of commands to execute
when a remote change occurs. This way I can trigger the mbsync -a command to synchronize my emails. Works like a charm! I will write a entry in the wiki about how to set up this kind of environment. We do not need to implement a cache system or an email synchronizer 🎉.

So why not turning Himalaya into a suite of PIMs?

I am thinking a lot about this question, and maybe the best is to create different repos. For example, about contacts (CardDAV), it could be very nice to have an external tool that allows us to manage contacts in a CLI-style way (like Himalaya), but also via a public lib. Himalaya could use this lib to complete contacts (if the feature is enabled).

[bligneri]

@srid I was not able to access my protonmail instance via bridge.
=> I would appreciate your help as you seem to have a similar setup as mine ;-)

I have an error even after having activated TLS via the option in the config file
11 imap-host = "127.0.0.1"
12 imap-port = 1143
13 imap-login = "cut & pasted from the config"
14 imap-passwd-cmd = "CUT & Pasted from the config"
15 imap-starttls = true

=> I have the following debug log (bot from he latest stable binary and also from the master build)

[2021-04-25T13:29:03 DEBUG himalaya] output format: PLAIN
[2021-04-25T13:29:03 DEBUG himalaya] log level: DEBUG
[2021-04-25T13:29:03 DEBUG himalaya::comp::cli] nothing matched
[2021-04-25T13:29:03 DEBUG himalaya] custom config path: Some("/Users/bligneri/.himalayarc")
[2021-04-25T13:29:03 DEBUG himalaya] init config
[2021-04-25T13:29:03 DEBUG himalaya] init account: default
[2021-04-25T13:29:03 DEBUG himalaya] mailbox: INBOX
[2021-04-25T13:29:03 DEBUG himalaya] begin matching
[2021-04-25T13:29:03 DEBUG himalaya::mbox::cli] nothing matched
[2021-04-25T13:29:03 DEBUG himalaya::flag::cli] nothing matched
[2021-04-25T13:29:03 DEBUG himalaya::imap::cli] nothing matched
[2021-04-25T13:29:03 DEBUG himalaya::msg::cli] default list command matched
[2021-04-25T13:29:03 DEBUG himalaya::imap::model] create TLS builder
[2021-04-25T13:29:03 DEBUG himalaya::config::model] insecure: false
[2021-04-25T13:29:03 DEBUG himalaya::imap::model] create client
[2021-04-25T13:29:03 DEBUG himalaya::config::model] STARTTLS: true
[2021-04-25T13:29:03 DEBUG himalaya::config::model] host: 127.0.0.1
[2021-04-25T13:29:03 DEBUG himalaya::config::model] port: 1143
[2021-04-25T13:29:03 DEBUG himalaya::imap::model] create session
[2021-04-25T13:29:13 ERROR himalaya] Cannot login to IMAP server
[2021-04-25T13:29:13 ERROR himalaya]  ↳ No Response: backend/credentials: incorrect password

[soywod]

@bligneri This is not the right place to report an issue (I moved it there: #129).

[ambroisie]

I would like to add that running dovecot locally just to enable offline access seems to me like a very heavy solution. I would like himalaya to be able to read emails that were fetched and stored using mbsync directly, instead of having a server constantly running on my laptop for this purpose.

Other than that, thank you for your work on this, I really like it :-)

[soywod]

I would like himalaya to be able to read emails that were fetched and stored using mbsync directly

This will be fixed by #43. Also, I planned to develop an IMAP synchronizer (when I find some free time), see #155 (comment).

[ambroisie]

That's good, I hadn't noticed that it was on the roadmap.

For my personal curiosity, why do you want to write your own version of isync and offlineimap, is it just personal curiosity?

[soywod]

It is just for the fact to be able to use cached messages within Himalaya (I planned to develop a CLI that is also a Rust lib). And also a bit for the curiosity 😉

[valera-rozuvan]

Everything is quite slow. This is because Himalaya does a request for every single action

Can we not do all the requests as parallel requests? Say, using p-threads or something?

I know that e-mail service providers can throttle, but I think they have some sane limits? Like maybe 5-10 parallel requests to their service will be allowed through?

[valera-rozuvan]

A possible solution would then be an interface service (daemon), which can run in the background and do periodic polling of 3rd party e-mail providers. It can then store results in memory/disk DB.

This way, the original himalaya software does not need modification. The user can choose to use a local interface for faster results, or the user can choose to query 3rd party e-mail servers directly.

The greatest benefit from this - the user will get immediate results from his queries. The interface will be queried by himalaya, and return results which are available immediately to the client. Also, the interface will initiate new requests to fetch new data in the background. The next time himalaya queries the interface, latest results will be fetched from the DB.

This can be a separate project altogether.

---

I am aware that this is similar to caching mechanism which was discussed earlier. However, if we separate this into a stand alone interface, himalaya can remain working even without this additional thing. It will be up to the user to choose how he wants his data fetched.

[soywod]

I thought about all this (there is an open issue on this daemon thing #156) and I agree with you. The best is to have a separate tool (a CLI and a lib) that synchronizes your emails. This way you can use it either via a terminal or a script (like offlineimap and mbysnc do), but I can also use it inside himalaya as a lib. I would also love to extend this concept to other aspect of emailing, like contacts and calendar. I see that there is a giant need, but I lack of time to do all of this 😞

[soywod]

(thanks a lot for you sponsoring, it really helps me a lot ❤️)

[valera-rozuvan]

I don't have large experience with Rust, and it would be very slow for me to undertake a large project in Rust ATM. However, I am very proficient with Node.js :)

Would you be willing to work together to define a set of contracts between himalaya and an interface daemon (I described above)? I could then write a reference implementation in Node.js of such a daemon.

I can also try and help out with changes to himalaya, PRs to add configuration and logic to enable working with an interface.

[soywod]

I like this idea of building a contract for this daemon, but I'm not so fan to do an implementation in Node.js. I also come from the Web world, and I'm convinced that Rust can bring a lot more in this domain than Node.js or Python. You know, this is not a urgent feature, you can take your time and see this as on opportunity to learn a new language 🙂 it is exactly what I did with himalaya, I learnt Rust through it and and I do not regret.

I started some POC around synchronizing contacts (CardDAV), and the contract is really simple in fact. There is only one function synchronize that takes 2 parameters: source_a and source_b. The most difficult part is to resolve conflicts.
Then this synchronize function can be called at the end of each himalaya command.