Implementing TiddlyWiki style atomic multi-category personal wiki, in Orgmode with deft/zetteldeft

[TRSx80]

I have been nibbling around the edges of this in my posts in #51 and #19, but rather than create an XY Problem I thought it might be helpful to explicitly state the actual use case I have in mind. So this is not an issue with Zetteldeft per say but rather a discussion of an idea amongst what I perceive to be a sort of group of like-minded folks. I hope that's alright.

I switched from TiddlyWiki to Orgmode for my personal notes / wiki / "commonplace book" (or whatever you like to call it) some years ago. There were a lot of reasons for that (which I will spare you) but the one key feature that I gave up was that atomic nature of individual, small "tiddlers" (as they are called in TiddlyWiki).

For those unfamiliar, a TiddlyWiki is essentially a self contained, editable HTML file, containing its own JavaScript microkernel and storing each "tiddler" internally as a separate <div>. Individual "tiddlers" are editable in a GUI in your browser in Markdown. You can connect the individual "tiddlers" in various ways by link, tag, etc. (all the things you would expect). Implementation difference aside, I think it's clear that, structurally, they are following similar philosophy to Zettelkasten.

But once I became familiar in Emacs/Org, there was just no going back. I also prefer to keep everything completely in plain text (and in separate files, instead of locked inside a big blob of JavaScript), much prefer Org to Markup (and there are some some killer features like moving table rows/columns around easily, which was always painful in Markdown), and if I'm being honest I never really cared for JavaScript anyway. Well, I said I was going to spare you (but apparently I can't help myself), so back to the point...

Ever since moving to Orgmode, I have been searching for a way to re-create this "atomic" structure like TiddlyWiki. Currently, my personal wiki.org is one big file, beginning from a large, single tree structure. Which works OK most of time but has it's downsides (mainly multi-categorization issue). I am sure I don't need to explain this to anyone looking into Zettelkasten method. However with deft (and likely Zetteldeft) I think I have finally found solution to this problem.

If this all goes well (and so far I have a good feeling), I have some ideas I am already thinking about implementing later on, like:

• automating back links (as discussed in Automated back links #51)
• automatically updating some metadata upon opening each note:
  • date/time last opened
  • increment count of times opened

Which would perhaps be interesting from a statistical analysis perspective (and even add to the overall knowledge/understanding perhaps, which I think is in line with Zettelkasten philosophy as I understand it currently), but not interesting enough to keep up with manually. Well at least that becomes a hassle and some times I forget. Much better to make it automatic.

Deft apparently provides deft-open-file-hook which is what I would plan on leveraging to implement these things (in concert with Orgmode properties API). Then something else maybe in the event the file was actually edited (other than metadata) where I would update another "last edited" property. Just thinking out loud here at this point.

I really do need to get my own blog going. Or maybe post this at Zettelkasten forums or something. I am just not sure where (and I already have an account here at GitHub).

Anyway, discussion is welcomed (as long as ELFS is OK with it here). 😄

Cheers! 🍻

[EFLS]

Welcome again and glad to hear your thoughts and ideas.

I try to keep Zetteldeft rather simple and basic, so I try to limit the number of core features. I'm not sure how your idea about metadata would fit this philosophy, but I'm am definitely open to expanding Zetteldeft (with hooks and what not) so that extensions like the one you suggest are easily implemented for users.

Automated back links would be a nice addition indeed, although much depends on what such a feature would do. I'm a little hesitant to simply include a link to any and all files that link to the current note, as this can easily become too much. More useful, I think, is to include a backlink when a new note is made from within another note. Anyway, such discussion is best reserved for #51 (or a separate issue).

With regards to the Zettelkasten method, the forum over at zettelkasten.de is often home to in-depth discussion. In my experience, however, the best way to get started is to start writing notes 😄 -- although I agree you need to explore available features first. Which reminds me that I should perhaps record a short video introducing Zetteldeft and how I use it to implement a Zettelkasten-like note taking system.

[TRSx80]

Thanks EFLS for tolerating my idiotic ramblings. 😄

I try to keep Zetteldeft rather simple and basic, so I try to limit the number of core features. I'm not sure how your idea about metadata would fit this philosophy, but I'm am definitely open to expanding Zetteldeft (with hooks and what not) so that extensions like the one you suggest are easily implemented for users.

I really appreciate your willingness to at least allow discussion, probably why I decided to share my thoughts here instead of somewhere else. I am just sort of "thinking out loud" about some ideas I had, not necessarily even trying to suggest that Zetteldeft is the right place for them. Maybe upstream in deft might be the correct place (although jblevins seems perhaps a bit "MIA" (busy elsewhere?) the last year or two)? Or even eventually in my own package / extension, or perhaps simply sharing the couple snippets I have come up with somewhere online (eg., in a gist). I am not sure how my ideas will continue to develop yet. So far it's just a few functions and some settings for (and small modification of) deft (although I do have more things in mind).

So far I implemented automatically updating metadata like the following. VISIT_* get updated every time you open the file, while EDIT_* only upon an actual edit, and subsequent save. As an added bonus, I hacked the deft code a little bit (couple lines in one place, which also fixed an existing bug) to allow updating the VISIT_* properties and then saving the file, without actually changing the file modification date, so as to not bring it to the top of the list just because you visited the file (assuming the default sort order which does that). Certainly I could make that behavior configurable though (need to think about / use it some more). Anyway, this is what it looks like currently (all of following was generated automatically just from entering the title in deft and then pressing <RET>):

* Test New Node
  :PROPERTIES:
  :EDIT_LAST: [2020-08-13 Thu 09:46]
  :EDIT_TOTAL: 1
  :PROP_VER: 0.1
  :VISIT_LAST: [2020-08-13 Thu 10:27]
  :VISIT_TOTAL: 4
  :END:
  :LOGBOOK:
  - State "CREATED"    from              [2020-08-13 Thu 09:46]
    - Created via deft.
  :END:

Above, I expanded it to demonstrate, but when you open a new node, the properties are collapsed, so it just looks like the following, nice and unobtrusive:

* Test New Node
  :PROPERTIES:...
  :LOGBOOK:...

This is just something I "always wanted to do" but I don't think I ever could before, until I found such an extensible tool as Emacs.

To be perfectly honest, thus far I have not actually implemented anything directly in/requiring zetteldeft, as I am still playing with functionality that seems to be included in the base deft.

I think I am also doing slightly different from the norm in deft, as my vision is each "node" being an actual Org heading, as opposed to using "#+TITLE: " type metadata. You gain a lot of advantages this way, namely Org tags, properties, etc. "for free" as can be seen above.

I am very happy with my progress so far, even if I am the only one using it. 😄 If anyone else is interested in this sort of workflow, let me know and I guess I can share the little bit of code I wrote so far.

Next things I am thinking about are:

• "automated" back links?
• browsing titles / opening these files on Android?

I am interested in why you would say the following about back links (maybe I just need to gain more experience in using the system):

I'm a little hesitant to simply include a link to any and all files that link to the current note, as this can easily become too much. More useful, I think, is to include a backlink when a new note is made from within another note.

... however I agree with you we can pick that up over in #51, in order to keep things organized. Or just discuss it here in the general sense (or in terms of what I am working on) as opposed to a possible feature of Zetteldeft. I do think it would be a lot easier to implement your suggested command, eg. "create a single two way link" rather than keeping "all links updated both ways automatically."

So, maybe not "automated" but "manual." Ultimately I think I need to keep using the system and gain experience and then I suppose I will see what you mean.

Which leaves the Android issue. Honestly something I had given up on, as (after trying for years to do anything productive) I have essentially given up on the platform and written it off as a "toy" operating system. However synchronizing plain text files across devices suddenly brings this back within the realm of possibility. And here is where I am currently conflicted when considering a file naming scheme.

Initially I had thought to keep the filenames as simply timestamps (eg. 2020-08-13_0830.org) as essentially a simple form of a UUID (to the second at least). This gives the advantage of allowing to change the title of any particular node, without breaking any already existing links. However then you would need some Android app which can read the first line of the file as a title, which is not something I have found yet (although the search continues). If anyone knows of such an app (preferably on F-Droid) please let me know. Another alternative would be automatically exporting from Org to HTML, which would make looking up information on mobile quite comfortable (assuming a proper reactive / mobile-first CSS) however would be read-only of course (which is probably fine for my use case, anyway).

The alternative is including the title directly in the file name. This has the advantage of being at first simpler, as you can simply open the files in any plain old text editor on Android, because now you can see their titles at least, right in the file name. However then keeping links up to date might become a hassle, especially in the case of changing the title of a particular node. I had thought that a package like Projectile (for instance) might be able to be leveraged in this case (Projectile already has a function which does "project wide" search and replace, which we could wrap, copy, or emulate (depending on what code is involved, I haven't looked at it) in order to make some sort of "change title" command. It can always be done manually as well of course (although less convenient). I suppose the salient question becomes "how often does one actually need to change the title of a particular node?"

I am sure I will have other thoughts as I go along, but this is where I am at presently.

Cheers!

[TRSx80]

I just came across this comment, which if I am understanding correctly (and nothing changed in the meantime) seems to indicate that combining date/time stamp with title in the file name is not even possible in base deft but rather something that has been implemented in zetteldeft due to this limitation.

I had not even noticed that yet. 😄 So, just making note...

[EFLS]

I think I am also doing slightly different from the norm in deft, as my vision is each "node" being an actual Org heading, as opposed to using "#+TITLE: " type metadata.

One thing to keep in mind is that both Deft and Zetteldeft are designed around the idea of "one note per file". But you are obviously free to do as you see fit 😄

Even though I don't think I'd use the metadata features myself, I am interested in seeing how you use this. Feel free to share details once your code and its use is a more practical workflow.

With regard to filenames: the Zetteldeft default tries to take these issues into account. The default setup includes both a (time-based) ID and the title of the note. This keeps files (relatively) legible outside of Emacs and Org. Moreover, the unique identifiers are used to find link destinations, so links are independent from filenames (as long as the ID is in the filename). Which is indeed, as you have noted in the mean time, the core of Zetteldeft.

As for mobile access, I think exporting to HTML is the most sensible option. Suggestions welcome in #69, especially with regards to extending the default org-publish to include links.

[TRSx80]

"one note per file"

Yes, of course, this is the whole point! 😄

I just think it is a good idea to (potentially) leverage all the functionality of Org (tags, metadata, properties, drawers, publishing, even TODOs potentially, and on and on...) by simply adding a "* " before the title on the first line (and saving it as .org file). Which are both easy to filter out for purposes of deriving a title, and then further automatically generating things like link/file names, etc... So much to gain for so little cost, IMO.

To put another way, imagine taking my current large tree structure, and chopping them all up into the smallest constituent bits. Each node remains a first level Org heading, within its own file. The first line / heading doubles as a title (as any other node/note in deft/zetteldeft). Then the structure can be recreated however you want (including multiple different ways) in intermediate structure nodes (and/or by many other methods, tags, etc.).

It appears to me @Dermody is thinking along the same lines insofar as leveraging Org, as he stated in his OP in #77.

But just because I want to potentially leverage Org functionality, in no way means I am abandoning the core concept of "one note per file." In fact, that was what my extended dissertation about TiddlyWiki was all about, above. This is the key (and only) feature I miss from that system, when I came over to Org. And this would give that back.

Perhaps this sort of implementation is not possible in zetteldeft, and that is why you are not imagining it. Or perhaps even simpler, maybe you are not an Org-mode user. 😄

So far I have only been playing with vanilla deft. But now that I have new information about the file naming possibilities, I think it is time for me to start playing with zetteldeft, if solely for that combination date/time stamp + title. That is indeed, a killer feature.

With regard to filenames: the Zetteldeft default tries to take these issues into account. The default setup includes both a (time-based) ID and the title of the note. This keeps files (relatively) legible outside of Emacs and Org. Moreover, the unique identifiers are used to find link destinations, so links are independent from filenames (as long as the ID is in the filename). Which is indeed, as you have noted in the mean time, the core of Zetteldeft.

It's clear you've thought this through, as we seem to be arriving at some of same conclusions. I guess next step for me is starting to become familiar with zetteldeft, especially file naming scheme (and in particular, combining date/time stamp + title) and how links are derived from that.

As for mobile access, I think exporting to HTML is the most sensible option. Suggestions welcome in #69, especially with regards to extending the default org-publish to include links.

This really would be the best option, at least for viewing, even if it would involve a little more work. Generally speaking, I tend to prefer the correct solution to the easy one. All in due time, I suppose...

[TRSx80]

Hi @EFLS,

Happy weekend! 🍻

I decided this is not the right place for this discussion (it probably never was).

So I made a post on r/Orgmode last night:

Implementing TiddlyWiki style atomic multi-category personal knowledge base / wiki, in Orgmode with deft? zetteldeft? notdeft? Actually thinking about rolling my own from parts of all the above.

I appreciate your tolerance to date 😄 and of course I would love to hear any feedback you might have as well, perhaps over there on my Reddit post. You are on Reddit, right? I seem to recall maybe reading Zetteldeft announcement there at some point.

In terms of this issue in the context of Zetteldeft, I suppose we can close it?

[EFLS]

Thanks, I am always happy when people discuss Zetteldeft on other forums, just to get the word out and generate some interest.

Let's indeed close this issue for now, but please do keep me posted about your journey.

[egh]

@TRSx80 To help myself understand how an org-mode based system would work, I forked the excellent https://github.com/EFLS/zd-tutorial as https://github.com/egh/zd-tutorial and explored some ideas there. You might be interested.

[TRSx80]

Hi, @egh,

Thanks for the heads up!

If I am understanding Zettelorg ID and link concepts correctly, it looks like you are using something like org-id or similar to implement long UUIDs?

I must admit, I never did like that approach. To me, the "killer feature" of Zetteldeft implementation is the ID/link implementation, such that those long unwieldy UUIDs are not necessary.

Although I do plan on sprinkling some more "native" type Org syntax on top, for example defining zdlinks as a new Org link type as described here.

Having said that, I do have some plans on adding :export and :store "methods" (for lack of a better term) in order to make zdlink protocol in Org a complete implementation.

Cheers!

[egh]

If I am understanding Zettelorg ID and link concepts correctly, it looks like you are using something like org-id or similar to implement long UUIDs?

Yes. I guess I can understand the dislike of UUID-based links, but the advantages are the good caching of links, and thus quick lookup. And you don't see the UUID when viewing in org mode. You can also modify org-id-method to change the form of IDs, e.g. to 'ts which uses timestamps.

[EFLS]

To help myself understand how an org-mode based system would work, I forked the excellent https://github.com/EFLS/zd-tutorial as https://github.com/egh/zd-tutorial and explored some ideas there. You might be interested.

That's impressive! I like how you overhauled both documentation and implementation. Is this Zettelorg a standalone package, or is it an experiment you explore within this set of notes & code blocks?

While it departs from the "everything should be human readable" idea of Zetteldeft, I like how you use Org IDs. When considering my options 2 years ago, I thought the caching of IDs would take too long to be convenient, but I might very well be mistaken. How is the performance?

[egh]

Thanks for taking a look!

Is this Zettelorg a standalone package, or is it an experiment you explore within this set of notes & code blocks?

It's just a throw away name for trying to figure out how to use org-mode only to implement a Zettelkasten style note system.

How is the performance?

Not bad - it takes a while to generate an id cache, but org maintains the cache as you add new notes, so as long as that is working (which it seems to), jumping to a link is very fast. Adding a new link can be slow if you are using the org-refile mechanism to browse all available headings.

org search can be a slow if you are searching using the built in tools, although org-occur-in-agenda-files is pretty fast.

[TRSx80]

You know what, when thinking just yesterday about "unrelated" issue of how I want to handle archiving completed tasks from my todo.org, I start to realize that org-id style long UUID might be most reliable way to keep track of where those things moved, especially in case they get moved to a different file (by year for example).

So I guess it really is all "related" after all, isn't it?

So maybe I need to re-think my stance on long UUID, or perhaps settle on different scheme as you suggested (timestamp based, similar to Zetteldeft).

I think I need to study org-id more, as it seems there are some performance reasons for using that hash. Also, I am curious how it resolves things that have "moved" (as in my archival case).

[egh]

I think I need to study org-id more, as it seems there are some performance reasons for using that hash. Also, I am curious how it resolves things that have "moved" (as in my archival case).

If you use org-refile the hash table storing locations is updated with the new location. Same with org-archive.

[TRSx80]

I suppose I should thank you @egh, as your innocuous comment just happened to intersect with me at precisely the right time, as I was already considering the "separate" issue of how I wanted to handle my workflow of moving some other "unrelated" things around (and still be able to reliably track their location).

Ultimately this led me to a chain reaction of thoughts whereby I slowly realized that the scope of what I would really like to do is probably outside (and/or much bigger than) what @EFLS probably has in mind for this project, and furthermore I would like to start from scratch with a "clean sheet" design of a full implementation of my own ideas.

The notion of Zettel will certainly be a part of it, but only one part. I am envisioning a complete, overarching system where each part handles what it is supposed to do (Zettel/Wiki, Archive, Journal, TODOs, even storage of binary files like photos, etc.) in the way most appropriate to their particular domains, and yet still have everything universally linked together with UUIDs and common, well defined file and other naming and tagging conventions, as well as being easily searchable by several different methods. In short, I want to be able to easily, quickly, and appropriately capture, organize, develop, publish, and archive all manner of thoughts, and, when needed, be able to recall any particular piece of information in a powerful and simple way. But maybe that is the goal we are all striving towards. You know, the ideal implementation of Vannevar Bush's Memex...

One of biggest questions I have is if such a thing is even needed, or if people just prefer creating their own custom workflows in Emacs/Org. I don't know yet, but I imagine that I will publish my vision of what such a system should look like, if for no other purpose than to advance the conversation we all seem to be having (in addition to using it myself, of course).

As a reward for EFLS tolerating all of my idiotic ramblings (and also because very few other people likely care 😄 ), once I decide on a name and get my preferred hosting space nailed down (which will probably not be GitHub) I will make the initial (private / soft opening) announcement here in this thread. So make a post and say hello if you want to receive notification whenever I do make that announcement, I suppose.

I have nothing (in principle) against contributing here in the future (and maybe we share some code / ideas eventually) however I expect all of my recent burning passion and energy to be poured now into this new project, as there is nothing quite like implementing your very own personal vision to provide motivation, as I am sure many of you will be able to relate.

With that, I will get out of your hair, for the time being at least... 😄

Cheers everybody, and thanks for all the great ideas and conversations! I really enjoyed all of them, and they helped me clarify the path I needed to take.

🍻

TRS-80

[egh]

Sounds interesting! Good luck with it :)

[EFLS]

what I would really like to do is probably outside (and/or much bigger than) what @EFLS probably has in mind for this project

I think so too. Zetteldeft's aim is providing a digital notebook with multiple hierarchies, not a system to encompass everything.

be able to recall any particular piece of information in a powerful and simple way.

Not sure whether you want even more inspiration, but if you know some Python, you should check out Karl Voit's Memacs. Part of it is a module that assigns timestamps to any file, as such providing a means for stable links to files even as they move around within your filesystem. Karl is also very approachable on Mastodon, so you could always try to contact him if you have direct questions. My only frustration is that he has some misconceptions with regards to Zettelkasten (as evidenced by his resent blog posts), but that's another matter.

Anyway, best of luck @TRSx80 and let us know how it goes.

[TRSx80]

Karl Voit

I have actually been a fan of his work for quite some years already. In fact, I already mostly implemented his sort of ISO-like file naming scheme, including his concept of tags as being separate from the description, controlled vocabularies, etc. His work has been very influential on my development of these sort of ideas, leading me to where I am today.

For whatever reason, I finally decided to reach out to him via email, and we already been having a nice correspondence so far. 👍