How to handle non-linear resources in the EPUB/HTML navigator?

[kerryknight]

Edit from @mickael-menu: This discussion (below) originally started on r2-navigator-swift, but soon broadened into a global conversation about the way Readium 2 should handle non-linear resources across platforms. Hence the transfer to architecture.

---

Hi! Thanks again for the great project and all your hard work!

I have what is hopefully a quick question: Is there a way to navigate to a non-linear TOC item?

SCENARIO:
I have a hyperlink in my main epub body that I want to tap to take me to an additional xhtml non-linear resource (item_body_previously_on idref listed below). This item is included in the EPUB table of contents. However, I can't figure out how to make this occur properly as the non-linear resource is getting skipped over when creating the publication's readingOrder and checking the publication'sreadingOrder seems to basically be at the root of every navigation interaction.

This functionality does not seem to work in r2-testapp-swift when I import my epub there as a test, either. However, the hyperlink does function correctly on r2-navigator-kotlin as well as when I import into Apple Books, so I don't think there's an issue with my epub itself. I have tried manually adding an additional Locator object for the non-linear resource to my publication's positionList to no avail.

Any quick advice or ideas on how I might be able to reach my desired functionality?

<spine toc="ncx">
    <itemref idref="item_body"/>
    <itemref idref="item_body_previously_on" linear="yes"/>
  </spine>

As the non-linear resource is in the publication's TOC, I've tried this below but with no success; it still skips over the resource on link clicks:

for resource in publication.tableOfContents {
  if publication.positionList.find({ (position) -> Bool in
    position.href == resource.href
  }) == nil {
    // need to add the TOC item to positionList
    let link = Link(href: resource.href, type: "application/xhtml+xml", title: "Previously On")
    var navigationLocation = Locator(link: link)
    navigationLocation.locations.progression = 0
    navigationLocation.locations.totalProgression = 0
    self.publication.positionList.append(navigationLocation)
                
    print("LINK: ADDED resource.href: \(resource.href)")
  }
}

Any and all advice is greatly appreciated!

[mickael-menu]

I don't think it's possible at the moment, since as you noticed readingOrder is queried before loading a resource.

Technically we could also load the document if it's listed under publication.resources, but then I wonder how is the pagination across chapters supposed to work if we have a non-linear item loaded in the web view? Do you know how iBooks and R2 Kotlin are handling this case?

Maybe we could talk about that on the next Readium call this Wednesday, if you're willing to join. See Weekly Meetings here.

Do you have a legal EPUB that we could use to test this?

[kerryknight]

Hi,

Thanks for the quick response. Unfortunately, I don't know exactly how the other platforms are handling it. I am by no means an Android dev, but this local setCurrent(resources:) function seems to diverge a bit from the Swift implementation. The lines where resourcePager.currentItem is set directly seem especially intriguing.

I'm including a .zip EPUB sample where we'd hoped to have the ability to click the hyperlink at the beginning of the EPUB that will load the additional included episode_previously_on.xhtml resource into view. The idea is that the additional resource is only accessible through this one hyperlink (we don't currently display an actual TOC) and the additional resource includes its own link back to the main content episode_body.xhtml. In practice, it would behave almost like that of a modal view, so that it would have no bearing on pagination through the main content. This feels like it should be a relatively common usage pattern overall, especially if we were to include promotional content in the EPUB only accessible under special circumstances/triggers, though we could certainly be wrong in how we're trying to make the EPUB behave overall.

Orphan Black: The Next Chapter - Our Needs To Shape Us: Part 1.epub.zip

Good idea on attending the call--I'll see if I can free up some time to make it.

Thanks again,
Kerry

[JayPanoz]

If that can help, @mickael-menu, I think (i)Books treats those links as something specific i.e. they are opened in a modal view without all the EPUB-specific stuff such as pagination, user settings, etc.

Actually, entire publications may even be designed around this assumption. So they are completely ignored when it comes to pagination, etc., they’re just displayed in a fullscreen modal and provide their own means of navigation. I can provide an EPUB privately if needed.

[mickael-menu]

Thanks for the sample @kerryknight, I could test with Apple Books on macOS and iOS, and as @JayPanoz described, it's presented as a modal without pagination nor style:

This follows a guideline from the spec:

The linear attribute allows Reading Systems to distinguish content that a user needs to access as part of the default reading order from supplementary content which might, for example, be presented in a popup window or omitted from an aural rendering.

On R2 Kotlin, the non-linear resource is loaded directly in the webview, but then we're stuck because there's no way to go back without a link inside the document, or through the table of contents.

Personally I'm not a fan of the modal view, because:

• We want to avoid an opinionated UX in the navigator.
• This doesn't work well with non-linear text adventures, which could make great use of non-linear chapters.

I suggest that:

• We load non-linear resources in the web view, as any reading order item, with user settings and internal pagination.
  • Or without pagination, if that would be too disruptive for books designed around a modal view. You have a particular example in mind @JayPanoz?
• We add a locator history in the testapp, to be able to go backward when jumping to a location.
  • This could almost already be implemented with locators, but we don't have a way to know for sure that a location change was from a disjointed jump, or a page turn.
  • Regardless of non-linear resources, I think it's a feature that Readium should have anyway.
• Optionally, we could add a shouldLoadNonLinearResource(Link) -> Bool delegate to allow reading app to intercept a non-linear resource, to be able to present it as a modal view manually.

Alternatively, we could delegate the presentation of non-linear resources to the reading apps altogether, by calling a presentResource(Link) function in the NavigatorDelegate.

[mickael-menu]

Here's another thread of people trying to use linear=no for non-linear adventure fictions: https://www.mobileread.com/forums/showthread.php?t=276216

Looks like Readium 1 was supporting this properly.

Some interesting notes from this post:

Other ereader programs support it, BUT:

• most of those handles the linear="no" like a big pop up, so often if you are in a linear="no" section and close the ebook and reopen it, you find yourself in the last linear="yes" section you read before enter the linear="no";
• some ereader programs handles the linear="no" like scroll pages instead paged ones (Ibooks desktop for example);
• some ebook reader programs do not allow the reader to turn the last page of a linear="no" section. Other allow and "fall back" in to the next linear="yes" section;

Also, while EPUB 3 readers may have failed these authors, we're supposed to be format agnostic in the navigator, so we can set a new precedent for webpub formats.

[llemeurfr]

I agree with the previous comments about the opportunity to allow a proper implementation of non-linear fictions.

This topic is UX oriented, i.e. joins authoring and reading system issues about presentation / navigation. It will be interesting to submit it to the W3C EPUB3 CG (soon W3C Publishing CG), as the current EPUB3 specification is clearly underspecified about the non-linear items and the discussion applies to Web Publications as well.

This community can therefore model what a non-linear fiction is and decide its practical characteristics (FXL/reflow resources, pagination/no, mandatory navigation links inside resources vs automatic navigation via path history, memory of the last position ...).

[HadrienGardeur]

IMO this covers multiple questions that we need to ask ourselves:

1. how do we restore your reading position after jumping to a locator?
2. how do we instantiate a navigator for resources outside the reading order?
3. how do we style such resources?

My personal take on this would be:

1. we need to emit an event that can be captured by the app when the user jumps to any locator or resource, which contains the locator of the current reading position
2. we've discussed this before and we need to draw a clear line between resources that are inside the publication (listed in resources) and those that are outside the publication (not listed in readingOrder or resources)
3. we should strictly style resources that are in EPUB and inside the bounds of a publication, any resource outside these boundaries or in another format (WPUB or RWPM for example) should not be styled

[llemeurfr]

@HadrienGardeur I agree that resources inside the bounds of a publication should have a specific treatment. IMO it should be true for any input format which is able to differentiate between what is inside and what is outside. EPUB is capable of that, WPUB and RWPM also have this notion because we explicitly defined readingOrder and resources as containers for what is "inside" the publication.

[HadrienGardeur]

@llemeurfr sure, the following things are specific to EPUB though:

• the concept of non-linear items (where something is in the reading order but shouldn't be reachable by following the reading order) doesn't exist in RWPM, so we move these resources to resources instead of readingOrder
• applying Readium CSS is only done on EPUB for now, although we could have a separate discussion about this (do we need a hint for that as well?)

[mickael-menu]

• the concept of non-linear items (where something is in the reading order but shouldn't be reachable by following the reading order) doesn't exist in RWPM, so we move these resources to resources instead of readingOrder

It implicitly exists though, through hyperlinks in HTML resources, or the table of contents for other formats such as audiobook. So we're faced with the same question: what happens when the user jumps from the ToC to an audio file that is not part of the reading order?

This also brings up some questions about the UX in reading apps: should the progress bar / page numbers be hidden when reading/playing a non-linear item?

[JayPanoz]

Or without pagination, if that would be too disruptive for books designed around a modal view. You have a particular example in mind @JayPanoz?

Yeah I have a few examples I can share – will try to do that via Slack today as I must retrieve these EPUBs first.

But what I can say is that at some point, at least in France, linear="no" was somewhat “popular” for more experimental publications from smaller publishers, because it’s not paginated in iBooks. They wanted to start with a blank page, as in browsers, but didn’t want to go Fixed-Layout – actually I’m not even sure you could reliably use FXL for that at the time so it may have been the only option for a while.

I guess you can say this use case is more of a work around, as there is no such thing as “give me a blank page” in EPUB, but EPUB was also the only reliable way for these publishers to make some money.

So that also made them iBooks’ exclusives in a way, given it was one of the very few but probably the biggest implementation of linear="no". And it also serves Apple’s own needs since they are using it themselves for tables and images (double-tap/click).

Which brings me to another use case I’m aware of: a ready to use modal where you can manage “widgets” e.g. image galeries, interactive maps, etc. more easily.

Pagination across RSs makes it incredibly hard to implement a modal yourself so it’s fair to say authors will try to leverage what already exists.

But once again, this use case kinda works because this linear="no" implementation doesn’t abide by the Reading App rules, it’s a blank canvas with the bare minimum UI (top bar with close button) + almost no gesture/event handling.

So on top of these “hacks” I guess there may well be vastly different use cases.

[mickael-menu]

Pagination across RSs makes it incredibly hard to implement a modal yourself so it’s fair to say authors will try to leverage what already exists.

Thanks for the details Jiminy, it looks like we opened a can of worms... :/

[kerryknight]

Wow, thanks @mickael-menu, et al. I'd never intended/expected this use case to garner this type of attention.

To give you a bit of more context on our use case, we produce our own content at my company and we're just now making the transition to utilizing EPUB3 from a standard HTML implementation. Currently, we have no plans to distribute our EPUB content outside our own platform (meaning our use case is not a standard one for Readium usage), so we're able to work around some of the aforementioned issues with our own set of hacks.

At present, we've had to utilize a native solution for the non-linear resource hyperlink where we capture the webview request url (which was skipped by Readium due to its lack of inclusion in readingOrder) to then open the content within a native modal view. While this works great now that it's implemented, it was indeed a bit more work on my end as it forced me to add a new webview in the modal view to load the non-linear content and then re-inject all my own CSS styling. So, to me, while the inclusion of a generic delegate callback shouldLoadNonLinearResource(Link) -> Bool would still be much more preferable to my hacky workaround, being able to handle it within the initial reader instance in some manner feels much better (assuming the existing stylesheets/user settings can be maintained within the non-linear content, at least as far as the user's font/theme selections go).

As for the "Should these modally-loaded, non-linear resources only scroll vertically or should they paginate?" argument goes, our special use case is completely fine with a standard vertical scroll only. However, if support were added out of the box, I could completely see us progressing further to JayPanoz' idea:

Which brings me to another use case I’m aware of: a ready to use modal where you can manage “widgets” e.g. image galeries, interactive maps, etc. more easily.

Thanks again, all!
-Kerry

[mickael-menu]

Currently, we have no plans to distribute our EPUB content outside our own platform (meaning our use case is not a standard one for Readium usage), so we're able to work around some of the aforementioned issues with our own set of hacks.

We've lived through the pains of upgrading private forks with Readium 1, so if we can provide a way to implement that kind of features without having you resorting to hacks, that would be best. Either as an extension point, or out of the box.

So maybe we need a way to declare what kind of presentation is expected for a non-linear resource (see this issue), to fit different use-cases: inline+paginated for text adventures, modal+scrollable for widgets.

To avoid breaking existing books using non-linear as modals with EPUB, but still work with non-linear fictions, we could:

• display non-linear resources full-screen, like any reading order item,
• restore last location even in non-linear resources,
• disable pagination and CSS styling in non-linear resources.

An author could always embed a RWPM inside an EPUB to override presentation hints to enable pagination and CSS styles in a non-linear resource, for example.

(Which makes me think that a tool to generate/synchronize a RWPM from an OPF could be useful to authors)

[mickael-menu]

We talked a bit more about this issue during last week's meeting. We agreed on:

• Adding a history of locations, through a "JumpedToLocator" event which will be fired when the navigator moves to a non-consecutive location.
  • The UX will be in the reading app, and we still need to figure out how to present it in an ergonomic way.
• Providing a way for reading apps to decide if a non-linear resource should be displayed paginated or not and with user styles or not.
  • This could potentially be chosen by the user in the app.
  • I think we should make this more generic by extending this hook for any resource, not just non-linear ones.

---

For the rendering of non-linear resources, the discussion we had on this related issue about footnotes can give us some clues.

The Navigator should have the simplest implementation – and in this case, also the correct one in my opinion: navigating to the non-linear resource in the main web view, like any resource. This will fire a LocatorChanged event, which means that the location can be restored.

However, we can add a hook to allow reading apps to intercept this navigation, e.g. navigator(shouldNavigateToResource:). If a reading app wants to implement a modal "à la iBooks", it can:

• Return false if the resource to be navigated to is non-linear, to prevent the Navigator from rendering it.
• Display a modal with a web view pointing to the resource's href.

Since in this case the Navigator won't navigate to the non-linear resource, then the last location stays the one from the current linear resource, which is compatible with iBooks' behavior.

[mickael-menu]

Here's a slide from the results of W3C EPUB Survey, which brings some light on this feature:

Do you use the linear property in the OPF spine?

38 of 65 (58%) use linear=“no”

• Hide content from the reader by eliminating it from the reading order, except if it’s requested on-demand or for internal use by the publication/reading system (e.g., repeated covers, navigation document, footnotes, answers to exercises, supplemental materials, etc.)
• Workaround for pop-up content and choose-your-own-adventure
• Use for dynamic advertising iframes that can be customized by the store

[danielweck]

Many good observations and analysis in this thread, just adding my own comments (copied from Slack). Sorry about the conversational style, I don’t have time to redact this properly tonight :)

in Readium1 we implemented support for non-linear spine items by displaying them just like any other HTML resources ... but the logic was incomplete because of "forking" from linear into ancillary resources was not handled properly, for example jumping back into linear reading order, previous/next navigation history, etc. (also, TTS and Media Overlays playback ... several ramifications)
other impact: annotations, search
(include / exclude ancillary documents?)
but I too am totally in favour of displaying non-linear documents with full support for most of not all features. it's mostly a question of letting the user know that jumping into ancillary resources leads to a cul-the-sac / dead-end, which requires navigating back into reading order
but note that there can be hyperlinks inside ancillary documents that lead to reading order ... or other non-linears ... it can get messy :)
example of problem to solve: save bookmark on non-linear spine item ... close app, reopen, follow bookmark, display ancillary document ... and then what? the originating context leading to the non-linear resource is not a previous spine item, so “back navigation” is not defined. previous/next buttons are disabled (well, work inside the paginated / scrolling non-linear document, but do not leave the resource)
none of this is defined by the specification, so it is not testable (user agent / reading system conformance requirement) ... implementations cannot be interoperable => content authors cannot rely on the feature.
note that i have not spent any serious dev time in Thorium/ReadiumDesktopSDK to solve this problem properly, from the top of my head I cannot say for sure where non-linear is processed

[danielweck]

See also:
https://docs.google.com/document/d/1b0jz9jpMGew2qj75m8RHklRi3FwafVRprieUDTzyX0E

Best place to reignite this can of worms :)
https://github.com/w3c/publ-epub-revision/issues

[mickael-menu]

Thanks for sharing your experience @danielweck,

but I too am totally in favour of displaying non-linear documents with full support for most of not all features. it's mostly a question of letting the user know that jumping into ancillary resources leads to a cul-the-sac / dead-end, which requires navigating back into reading order

Seems like reading app concern to me, but we can certainly provide best practices. A documentation addressing this would be useful. I think that showing a "back" history icon is probably the most common way of doing this, and useful beyond this particular use case. It would also work with non-linear fictions because not too intrusive (e.g. paginating back to the next linear item at the end of the non-linear resource would break such fictions).

example of problem to solve: save bookmark on non-linear spine item ... close app, reopen, follow bookmark, display ancillary document ... and then what? the originating context leading to the non-linear resource is not a previous spine item, so “back navigation” is not defined. previous/next buttons are disabled (well, work inside the paginated / scrolling non-linear document, but do not leave the resource)

That can prove challenging... Assuming the publication is properly designed, there's a link in the non-linear resource to go somewhere else, or a table of contents to jump to another resource.

But in the reading app, I can think of two ways of addressing this:

• With an actionable progress bar (or similar) to jump to another location. This is not great because we end up wherever.
• Adding and persisting a (or a list of) "referrer" locator alongside the bookmark to restore some history context. This is a bit weird for regular bookmarks, but would solve the "last location restoration" issue. I can see this being useful for linear resources as well.

See also:
https://docs.google.com/document/d/1b0jz9jpMGew2qj75m8RHklRi3FwafVRprieUDTzyX0E

Nice! Thanks for sharing

[mickael-menu]

A related issue with a sample book was opened by @civodulab on the Kotlin repo: readium/kotlin-toolkit#124