Orgmode in Emacs 30+ breaks org-next-visible-heading

[chewxy]

What Is Going On

Consider an org file that looks as follows:

* Heading 1
(not (org-next-visible-heading 1)) ;; FIRST
some content for Heading 1
** Heading 2a 
(not (org-next-visible-heading 1)) ;; SECOND A
some content for Heading 2a
** Heading 2b
(not (org-next-visible-heading 1)) ;; SECOND B
some content for Heading 2b
* Heading 3
(not (org-next-visible-heading 1)) ;; THIRD
some content for Heading 3

When you eval the sexp at FIRST, the point jumps to the head of Heading 2a. Then when you eval the sexp at SECOND A, the point jumps to the head of Heading 2b, and so on and so forth.

So far so good. But not so if you are relying on the return value of org-next-visible-heading.

In my older machine with Org 9.6.1 on Emacs 29, the return value of FIRST, SECOND A, SECOND B are t and the return value of THIRD is `nil.

In my newer machines with Org 9.7.9 on Emacs 30, the return values are all nil.

Why This Matters

Currently there are 31 instances of (while (not (org-next-visible-heading 1)) or (when (not (org-next-visible-heading 1)) in Org-Novelist. All these loops are broken in Org 9.7.9.

These loops are used to gather headings for export or indicing purposes - what they do is they iterate through all the headings until a heading matches the given subheading.

Consequently, when a function like org-novelist-update-references is called (either by hook or manually invoked), you will get the Glossary simply appended, instead of updated.

Example 1: Improperly updated references.

Consider a chapter that looks something like this:

; -*-Org-Novelist-*-
#+TITLE: Begin At The Beginning
[[file:../Notes/chapter-BeginAtTheBeginning-notes.org][Notes]] are available for this [[file:../Indices/chapters.org][chapter]] from [[file:../main.org][My Story]].
* Content
Supply Some Spicy Scenes here!
* Glossary
** Characters
*** <<<Protagonist>>> View Notes
*** <<<Antagonist>>> View Notes 
** Places
*** <<<Universe>>> View Notes 
** Props
*** <<<People Use Props?>>> View Notes

In Emacs 30, when you run org-novelist-update-references , what will get generated is the following:

; -*-Org-Novelist-*-
#+TITLE: Begin At The Beginning
[[file:../Notes/chapter-BeginAtTheBeginning-notes.org][Notes]] are available for this [[file:../Indices/chapters.org][chapter]] from [[file:../main.org][My Story]].
* Content
Supply Some Spicy Scenes here!
* Glossary
** Characters
*** <<<Protagonist>>> View Notes
*** <<<Antagonist>>> View Notes 
** Places
*** <<<Universe>>> View Notes 
** Props
*** <<<People Use Props?>>> View Notes
* Glossary
** Characters
*** <<<Protagonist>>> View Notes
*** <<<Antagonist>>> View Notes 
** Places
*** <<<Universe>>> View Notes 
** Props
*** <<<People Use Props?>>> View Notes

Note the repeated Glossary headings

Example 2: Inability to export stories

Because there exist this snippet of code in org-novelist-export-story:

 (orgn--fold-show-all)  ; Belts and braces
                  (while (not (org-next-visible-heading 1))
                    (when (string= (orgn--ls "front-matter-heading") (nth 4 (org-heading-components)))
                      ;; Found front matter, get files in order and add to list.
                      (when (org-goto-first-child)
                        (setq fm-file-list (cons (orgn--heading-last-link-absolute-link-text) fm-file-list))
                        (while (org-goto-sibling)
                          (setq fm-file-list (cons (orgn--heading-last-link-absolute-link-text) fm-file-list))))
                      (goto-char (point-max))))  ; No need to check any more, so skip to the end go exit loop

and because (not (org-next-visible-heading 1)) will return nil in Emacs 30, none of the front-matter-heading stuff will ever get matched, thus nothing gets exported.

Potential Solution

While I have not examined in detail each of the 31 instances of the use of (not (org-next-visible-heading 1)), I suspect most of them being in a loop is meant to fulfil the purpose of finding a particular subheading. I suspect the most prudent thing to do would be to abstract these instances into one or two functions that doesn't rely on the dodgy return value of org-next-visible-heading

I haven't quite the time to fix this right now, but I may have time after November (after GopherConAU)

[chewxy]

OK I lied (about not having quite the time). Here's a prototype I hacked up. The quality isn't the best as I had my attention diverted a few times

(defun wrangle (heading replacement)
  (save-excursion
    (goto-char (point-min))
    (goto-char (org-find-exact-headline-in-buffer heading (current-buffer) t)) ;; TODOMAYBE use regex
    (forward-line)
    (message "found exact headline. point is %s" (point))
    (let ((start (point))
          (end (progn (outline-next-heading) (point))))
	 (message "delete beteen  %s and %s" start end)
         (let ((content (buffer-substring-no-properties start end)))
	      (delete-region start end)
	      (goto-char start)
	      (insert (if (functionp replacement)
	                  (funcall replacement content)
			  replacement))
	       (insert "\n")))))

Given a similar org file as above:

* Heading 1
XXX
** Heading 2a 
YYY
** Heading 2b
ZZZ
* Heading 3
AAA

Here are two ways to use wrangle:

1. Replacing the contents with a constant string (presumably constructed elsewhere, much like how update-references does it:

(wrangle "Heading 2a" "HAHA")

This will yield

* Heading 1
XXX
** Heading 2a 
HAHA
** Heading 2b
ZZZ
* Heading 3
AAA

2. Replacing the contents with a function that takes the current content into account:

(wrangle "Heading 2b" (lambda (x) (if (string= "ZZZ\n" x) "BBB" "CCC")))

This will yield

* Heading 1
XXX
** Heading 2a 
YYY
** Heading 2b
BBB
* Heading 3
AAA

What is Not Done

I have not added any kind of error handling so

(wrangle "Invalid Heading" "Foo")

will yield an error. and no effect.

It's getting late though (11pm here) so I leave the rest to you to figure out

[sympodius]

Thanks for spotting this!

This does seem like something that I'll need to create a proper solution for, but in the meantime I'm duplicating the older working version of org-next-visible-heading and placing it into an internal function, org-novelist--next-visible-heading. This seems the solution least likely to break things until better code can be created.