Files created by org-journal are not picked up by org-roam

[DominikMendel]

I recently have been wrestling with my org-journal files and dynamically adding all my org-roam files with TODOs in them to be included into my org-agenda-files, including my org-journal files that are in a sub directory in org-roam. I have the exact implementation from (https://gist.github.com/d12frosted/a60e8ccb9aceba031af243dff0d19b2e) implemented. The adding and removing of the tags works great. I have an issue with getting the tags from vulpea-project-files to go into my org-agenda-files.

The provided function:

(defun vulpea-project-files ()
    "Return a list of note files containing 'project' tag." ;
    (seq-uniq
     (seq-map
      #'car
      (org-roam-db-query
       [:select [nodes:file]
        :from tags
        :left-join nodes
        :on (= tags:node-id nodes:id)
        :where (like tag (quote "%\"project\"%"))]))))

And the supporting functions that call it :

(defun vulpea-agenda-files-update (&rest _)
  "Update the value of `org-agenda-files'."
  (setq org-agenda-files (vulpea-project-files)))

(add-hook 'find-file-hook #'vulpea-project-update-tag)
(add-hook 'before-save-hook #'vulpea-project-update-tag)

(advice-add 'org-agenda :before #'vulpea-agenda-files-update)
(advice-add 'org-todo-list :before #'vulpea-agenda-files-update)

I am going to try and provide everything I can about the org file.
Contents :
Image since GitHub format the text odd:

Major mode is org-journal, as I created this file from calling org-journal's org-journal-new-scheduled-entry.
Absolute file path: ~/Dropbox/org/roam/daily/Personal 2022-02-23.org
Maybe there is some issue with there being a space, or dashes?
I have verified the file is found by org-roam-node-find.

My attempts to use different functions for vulpea-project-files that yielded the same result.

(defun vulpea-project-files ()
  "Return a list of note files that are part of `org-agenda'."
  (seq-map
   #'vulpea-note-path
   (vulpea-db-query-by-tags-every '("project"))))

(defun vulpea-project-files ()
  "Return a list of note files that are part of `org-agenda'."
  (seq-map
   #'vulpea-note-path
   (vulpea-db-query-by-tags-some '("project"))))

All 3 functions are not finding that org file for some reason.

Any help here would be great. I feel like I am so close to achieving what I want, and this is the last hi-cup.

[d12frosted]

Answer to this question is located in comments.

---

Hey,

Everything seems fine. Especially if you can org-roam-node-find it 😸 Basically, the first direct org-roam-db-query should work because it doesn't rely on setting up anything extra. Unlike vulpea functions that require some setup and I don't know if you did it properly. So just to narrow us, can you please share result of the following scripts?

(org-roam-db-query
 [:select *
  :from tags
  :where (= node-id "d147530a-294c-4bcc-ad69-dec962462869")])

(org-roam-db-query
 [:select [id file title]
  :from nodes
  :where (= id "d147530a-294c-4bcc-ad69-dec962462869")])

[d12frosted]

That's very strange. Because somehow org-roam syncs your files later on. I will try to setup org-journal myself today to double check that mode deriving is not messing around.

[d12frosted]

Just tried to setup org-journal. For reference, this is my org-journal setup (IDs are propagated using vulpea-id functionality):

(use-package org-journal
  :defer t
  :init
  (setq-default
   org-journal-dir (expand-file-name "org-journal/" org-roam-directory)
   org-journal-file-format "%Y%m%d.org"
   org-journal-file-header "#+title: %Y%m%d\n"))

And it works on my end. The only hiccup that I faced - after-save-hook and before-save-hook are messed when org-journal creates a new entry. They are fixed by re-visiting file though (e.g. save buffer, kill buffer, find file again). And this is why synchronisation doesn't work.

So TL;DR - the problem is with messed up hooks. Trying to patch them by manually saving file, updating db is not going to lead you anywhere. You are fixing symptoms instead of the real issue.

Attempt to debug

I tried to debug it a little bit, but could not understand at what point it becomes corrupted. For example, I see that org-roam-db-autosync--setup-file-h is being called and that it modifies after-save-hook from (1) to (2) by appending org-roam-db-autosync--try-update-on-save-h:

(1): (vino-db-update-file flycheck-handle-save diff-hl-update org-journal-after-save-hook t ws-butler-after-save)
(2): (org-roam-db-autosync--try-update-on-save-h vino-db-update-file flycheck-handle-save diff-hl-update org-journal-after-save-hook t ws-butler-after-save)

But then somehow after-save-hook gets the following value (e.g. org-roam-db-autosync--try-update-on-save-h and vino-db-update-file are dropped):

(flycheck-handle-save diff-hl-update org-journal-after-save-hook t ws-butler-after-save)

Workaround

Obviously, you can restore these hooks in org-journal-after-entry-create-hook:

(add-hook 'org-journal-after-entry-create-hook #'org-roam-db-autosync--setup-file-h)
(add-hook 'org-journal-after-entry-create-hook #'org-journal-restore-hooks)

(defun org-journal-restore-hooks ()
  (add-hook 'find-file-hook #'vulpea-project-update-tag)
  (add-hook 'before-save-hook #'vulpea-project-update-tag))

But overall, it's a bad workaround, because it fixes only stuff that I noticed to be broken. Other things might be affected by this 'corruption'.

At least...

Well, at least it means that you can simplify your org-journal-after-entry-create-hook and get rid of org-create-new-id-journal, because you don't need to call org-roam-db-update-file manually:

(add-hook 'org-journal-after-header-create-hook #'org-id-get-create)
(add-hook 'org-journal-after-entry-create-hook #'org-roam-db-autosync--setup-file-h)
(add-hook 'org-journal-after-entry-create-hook #'org-journal-restore-hooks)

(defun org-journal-restore-hooks ()
  (add-hook 'find-file-hook #'vulpea-project-update-tag)
  (add-hook 'before-save-hook #'vulpea-project-update-tag))

And then once you save the buffer, everything should be synced.

Real fix?

I don't have it. And at this point I don't understand what exactly breaks these values. I need to read org-journal code to better understand what's happening there.

And in any case, these issues are not related to vulpea at all, but rather to either Emacs itself or combination of org-roam and org-journal. I don't want to discourage your from asking questions here or via Email, just want to keep discussions in this repo relevant for vulpea. These two big threads that we had are not related to vulpea and both related to one single issue - hooks corruption when org-journal is used together with org-roam. Of course, initially it was not clear, but now that it's clear 😸 Again, don't hesitate to write me on this topic via Email. Also, there is org-roam discousure, where you can ask any questions related to org-roam and many people can help you there. If you want to ping me there, just use my GitHub handle 😸

And at this point I have another question - why do you want to use org-journal instead of journaling capabilities provided by org-roam? 😸

Answers to your questions

Still feel like I need to comment on your questions from your previous message.

I did a little more digging and noticed in vulpea-project-update-tag you call vulpea-buffer-tags-set to actually set the tag. This looks like you are manually adding text to a file, I wonder if it would be better to somehow use org-roam-tag-add and maybe that will fix my issue? This however I believe only sets a single tag at a time, so I am not experienced enough to know what to add to test this change.

Both org-roam-tag-add and vulpea-buffer-tags-set are pretty much the same. They simply modify the buffer' content. In my code I am using vulpea functions for two reasons: (1) #+filetags was not supported by org-roam API at some point, so I had to write some helpers and (2) org-roam API was changing many times and I wanted something more stable 😸

I also noticed roam has a org-roam-db-insert-tags function it calls in the org-roam-db-update-file which is what I started using in my org-create-new-id-journal like you recommended. Maybe I need to add org-roam-db-update-file or org-roam-db-insert-tags myself somewhere in the vulpea functions? I don't know anymore, this should all be happening in my save hooks so I don't know why this is happening. Kinda grasping at straws here.

I recommend against calling org-roam-db-update-file. I just meant that you don't need to sync everything and instead you can use more narrowed function that does what you want. But again, I was pointing out that save-buffer should do that for yourself and you don't need to mess with db manually.

Maybe I need to add org-roam-db-update-file or org-roam-db-insert-tags myself somewhere in the vulpea functions?

No 😸 These functions modify buffer and that's it. Unless they are interactive they should not modify any other state. The idea behind org-roam sync routine is that it happens when you save the buffer, e.g. when you explicitly say 'yes, now this is a snapshot that I want. Unfortunately, sync is not free and it needs to occur at some sane points. Since vulpea` functions we are talking about are usually used programatically, they are sometimes used in batch, and adding unnecessary db syncs in them would mean performance degradation, and unfortunately, performance is very critical for me.

I hope that gives some 'philosophical' background for what's happening here 😸

[DominikMendel]

I greatly appreciate all the work you have spend into this. It was very helpful. I am glad we were able to narrow this issue down to something from org-journal. I am sorry that this deviated from being a Vulpea discussion.

Everything you explained also made perfect sense, so thanks again for that.

(add-hook 'org-journal-after-header-create-hook #'org-id-get-create)
(add-hook 'org-journal-after-entry-create-hook #'org-roam-db-autosync--setup-file-h)
(add-hook 'org-journal-after-entry-create-hook #'org-journal-restore-hooks)

(defun org-journal-restore-hooks ()
  (add-hook 'find-file-hook #'vulpea-project-update-tag)
  (add-hook 'before-save-hook #'vulpea-project-update-tag))

This worked well.

I think I am going to keep your recommended hooks in for now, as I don't see anything wrong with using them and they work fine. Like you said though, this isn't fixing the main issue. I might still communicate with you in the future about some potential fixed with org-journal if you don't mind.
I have a current open bug for org-journal open here bastibe/org-journal#382 (comment). It appears that this is just a known issue that they haven't fixed and might not.

You asked "And at this point I have another question - why do you want to use org-journal instead of journaling capabilities provided by org-roam?"
I use "both". Mostly I use org-journal for some of the very nice helpful functions that provide. But really everything is just org-roam journals. Functions like their journal carry over TODOs.

[d12frosted]

It was very helpful.

Glad to hear 😸

I am sorry that this deviated from being a Vulpea discussion.

No worries.

This worked well.

🎉

I might still communicate with you in the future about some potential fixed with org-journal if you don't mind.

Please do 😸

I have a current open bug for org-journal open here bastibe/org-journal#382 (comment). It appears that this is just a known issue that they haven't fixed and might not.

Thanks for the link. Subscribed.

Mostly I use org-journal for some of the very nice helpful functions that provide. But really everything is just org-roam journals. Functions like their journal carry over TODOs.

But why won't you create journal entries via org-roam?

[DominikMendel]

But why won't you create journal entries via org-roam?

There are 2 functions I use that are specific to org-journal.

1. org-journal-new-entry. Which I use less often but I still occasionally use.
2. The next thing I use is for my org-capture-templates they provided a function to find the date of the function. I find this very useful, I am unsure how dependent this really is on the org-journal package.

Functions used

(defvar org-journal--date-location-scheduled-time nil)

(defun org-journal-date-location (&optional scheduled-time)
  (let ((scheduled-time (or scheduled-time (org-read-date nil nil nil "Date:"))))
    (setq org-journal--date-location-scheduled-time scheduled-time)
    (org-journal-new-entry t (org-time-string-to-time scheduled-time))
    (unless (eq org-journal-file-type 'daily)
      (org-narrow-to-subtree))
    (goto-char (point-max))))

(defun org-journal-find-location ()
  ;; Open today's journal, but specify a non-nil prefix argument in order to
  ;; inhibit inserting the heading; org-capture will insert the heading.
  (org-journal-new-entry t)
  (unless (eq org-journal-file-type 'daily)
    (org-narrow-to-subtree))
  (goto-char (point-max)))

Using them in an org template

Inside of a (setq org-capture-templates):

          ("jt" "Journal New Clock Entry" plain (function org-journal-find-location)
           "** %(format-time-string org-journal-time-format)%?"
           :jump-to-captured nil
           :immediate-finish nil
           :clock-in t)

For me the high appeal of org-journal is if you have outstanding TODOs in an older date, creating a new journal file carries over the TODOs. I believe this is done with org-journal-carry-over-items and org-journal-handle-old-carryover.