[BUG] koreader sync not working

[LnLcFlx]

First of all, thanks a lot for implementing this feature!

Description of the Bug

I followed https://github.com/stumpapp/stump/blob/4055495caf7ce3fcedca14a484fdc73aa32531c0/docs/pages/guides/integrations/koreader.mdx#koreader-sync on how to set up koreader sync.
I am running koreader version 2024.07-8802 on a Tolino Vision 5.
When I click on "Push progress from this device now" I get "Something went wrong syncing progress, check your network connection and try again later".

I used an arbitrary username and password as it wouldn't let me use an empty one, but it says the login was successful.

Yes, I have

• ENABLE_KOREADER_SYNC=true
• KoReader-compatible hashes enabled
• tried to turn it off and on again

Build Details:

• Version: 0.0.7, unstable image, 71294ab
• Docker: yes

[aaronleopold]

Thanks for reporting this issue!

First I just want to double check that when you say "unstable image" that you are referring to the experimental image? The commit you specified looks to be right, but there is an old unstable that should probably be removed now.

Something went wrong syncing progress, check your network connection and try again later

If you can bump the verbosity and try again, please share the logs to see if perhaps something helpful can be found from there.

KoReader-compatible hashes enabled

Can you confirm there are hashes being generated? I'm not sure how comfortable you are with SQLite browsers, but you can just query for a media record you expect to have one and check that the koreader_hash column is set. I will say that if you set the field on an existing library, it probably isn't set. The TLDR; for that is some remaining work to improve the scanner to granularly opt-in to some operations to "force" a visit to a particular book / set of books. It was already started with a small set of options, and while it is exposed to the API the UI does not yet surface it. So you can use the API directly to make those hashes get set, if needed

I'm just getting back from DC for a work trip, so won't be too responsive the rest of the night. However I'll try and dedicate some time this weekend to debug further

[LnLcFlx]

Thank you for the quick reply.

I was indeed using unstable but on experimental I get the same error.

For clarification: The error message, I received in a Koreader pup-up.

Indeed, in stump.db in the media table koreader_hash is NULL.
But I can confirm generate_koreader_hashes in library_config is true.
I also rescanned my libraries.

Using STUMP_VERBOSITY=3 I get a 404 for /:api-key/syncs/progress in the logs:

stump  |   2024-11-16T09:36:44.870312Z DEBUG tower_http::trace::on_request: started processing request
stump  |     at /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/tower-http-0.5.2/src/trace/on_request.rs:80
stump  |     in tower_http::trace::make_span::request with method: PUT, uri: /koreader/stump_REDACTED/syncs/progress, version: HTTP/1.1
stump  | 
stump  |   2024-11-16T09:36:44.870969Z DEBUG quaint::connector::metrics: query: SELECT `main`.`api_keys`.`id`, `main`.`api_keys`.`name`, `main`.`api_keys`.`short_token`, `main`.`api_keys`.`long_token_hash`, `main`.`api_keys`.`permissions`, `main`.`api_keys`.`created_at`, `main`.`api_keys`.`last_used_at`, `main`.`api_keys`.`expires_at`, `main`.`api_keys`.`user_id` FROM `main`.`api_keys` WHERE (`main`.`api_keys`.`short_token` = ? AND `main`.`api_keys`.`long_token_hash` = ? AND (`main`.`api_keys`.`expires_at` >= ? OR `main`.`api_keys`.`expires_at` IS NULL) AND (`main`.`api_keys`.`id`) IN (SELECT `t0`.`id` FROM `main`.`api_keys` AS `t0` INNER JOIN `main`.`users` AS `j0` ON (`j0`.`id`) = (`t0`.`user_id`) WHERE (`j0`.`deleted_at` IS NULL AND `j0`.`is_locked` = ? AND `t0`.`id` IS NOT NULL))) LIMIT ? OFFSET ?, params: ["REDACTED","c6d6a85b425938b8b4b46ba06d18fa9b9479a1716ac2ed6d05c1fe99a25c18de","2024-11-16 09:36:44.870397082 UTC",false,1,0], result: "success", item_type: "query", is_query: true, duration_ms: 0
stump  |     at /usr/local/cargo/git/checkouts/quaint-453f07acedd929b5/c502995/src/connector/metrics.rs:87
stump  |     in quaint::connector::metrics::quaint:query with db.statement: SELECT `main`.`api_keys`.`id`, `main`.`api_keys`.`name`, `main`.`api_keys`.`short_token`, `main`.`api_keys`.`long_token_hash`, `main`.`api_keys`.`permissions`, `main`.`api_keys`.`created_at`, `main`.`api_keys`.`last_used_at`, `main`.`api_keys`.`expires_at`, `main`.`api_keys`.`user_id` FROM `main`.`api_keys` WHERE (`main`.`api_keys`.`short_token` = ? AND `main`.`api_keys`.`long_token_hash` = ? AND (`main`.`api_keys`.`expires_at` >= ? OR `main`.`api_keys`.`expires_at` IS NULL) AND (`main`.`api_keys`.`id`) IN (SELECT `t0`.`id` FROM `main`.`api_keys` AS `t0` INNER JOIN `main`.`users` AS `j0` ON (`j0`.`id`) = (`t0`.`user_id`) WHERE (`j0`.`deleted_at` IS NULL AND `j0`.`is_locked` = ? AND `t0`.`id` IS NOT NULL))) LIMIT ? OFFSET ?
stump  |     in tower_http::trace::make_span::request with method: PUT, uri: /koreader/stump_REDACTED/syncs/progress, version: HTTP/1.1
stump  | 
stump  |   2024-11-16T09:36:44.871087Z DEBUG quaint::connector::metrics: query: SELECT `main`.`users`.`id`, `main`.`users`.`username`, `main`.`users`.`hashed_password`, `main`.`users`.`is_server_owner`, `main`.`users`.`avatar_url`, `main`.`users`.`last_login`, `main`.`users`.`created_at`, `main`.`users`.`deleted_at`, `main`.`users`.`is_locked`, `main`.`users`.`max_sessions_allowed`, `main`.`users`.`permissions`, `main`.`users`.`user_preferences_id` FROM `main`.`users` WHERE `main`.`users`.`id` IN (?) LIMIT ? OFFSET ?, params: ["clshhdlwn0002pu11li2922u8",-1,0], result: "success", item_type: "query", is_query: true, duration_ms: 0
stump  |     at /usr/local/cargo/git/checkouts/quaint-453f07acedd929b5/c502995/src/connector/metrics.rs:87
stump  |     in quaint::connector::metrics::quaint:query with db.statement: SELECT `main`.`users`.`id`, `main`.`users`.`username`, `main`.`users`.`hashed_password`, `main`.`users`.`is_server_owner`, `main`.`users`.`avatar_url`, `main`.`users`.`last_login`, `main`.`users`.`created_at`, `main`.`users`.`deleted_at`, `main`.`users`.`is_locked`, `main`.`users`.`max_sessions_allowed`, `main`.`users`.`permissions`, `main`.`users`.`user_preferences_id` FROM `main`.`users` WHERE `main`.`users`.`id` IN (?) LIMIT ? OFFSET ?
stump  |     in tower_http::trace::make_span::request with method: PUT, uri: /koreader/stump_REDACTED/syncs/progress, version: HTTP/1.1
stump  | 
stump  |   2024-11-16T09:36:44.871255Z DEBUG quaint::connector::metrics: query: BEGIN, params: [], result: "success", item_type: "query", is_query: true, duration_ms: 0
stump  |     at /usr/local/cargo/git/checkouts/quaint-453f07acedd929b5/c502995/src/connector/metrics.rs:87
stump  |     in quaint::connector::metrics::quaint:query with db.statement: BEGIN
stump  |     in tower_http::trace::make_span::request with method: PUT, uri: /koreader/stump_REDACTED/syncs/progress, version: HTTP/1.1
stump  | 
stump  |   2024-11-16T09:36:44.871379Z DEBUG quaint::connector::metrics: query: SELECT `main`.`api_keys`.`id` FROM `main`.`api_keys` WHERE (`main`.`api_keys`.`id` = ? AND 1=1), params: [2], result: "success", item_type: "query", is_query: true, duration_ms: 0
stump  |     at /usr/local/cargo/git/checkouts/quaint-453f07acedd929b5/c502995/src/connector/metrics.rs:87
stump  |     in quaint::connector::metrics::quaint:query with db.statement: SELECT `main`.`api_keys`.`id` FROM `main`.`api_keys` WHERE (`main`.`api_keys`.`id` = ? AND 1=1)
stump  |     in tower_http::trace::make_span::request with method: PUT, uri: /koreader/stump_REDACTED/syncs/progress, version: HTTP/1.1
stump  | 
stump  |   2024-11-16T09:36:44.871433Z DEBUG quaint::connector::metrics: query: UPDATE `main`.`api_keys` SET `last_used_at` = ? WHERE (`main`.`api_keys`.`id` IN (?) AND (`main`.`api_keys`.`id` = ? AND 1=1)), params: ["2024-11-16 09:36:44.871190831 UTC",2,2], result: "success", item_type: "query", is_query: true, duration_ms: 0
stump  |     at /usr/local/cargo/git/checkouts/quaint-453f07acedd929b5/c502995/src/connector/metrics.rs:87
stump  |     in quaint::connector::metrics::quaint:query with db.statement: UPDATE `main`.`api_keys` SET `last_used_at` = ? WHERE (`main`.`api_keys`.`id` IN (?) AND (`main`.`api_keys`.`id` = ? AND 1=1))
stump  |     in tower_http::trace::make_span::request with method: PUT, uri: /koreader/stump_REDACTED/syncs/progress, version: HTTP/1.1
stump  | 
stump  |   2024-11-16T09:36:44.871540Z DEBUG quaint::connector::metrics: query: SELECT `main`.`api_keys`.`id`, `main`.`api_keys`.`name`, `main`.`api_keys`.`short_token`, `main`.`api_keys`.`long_token_hash`, `main`.`api_keys`.`permissions`, `main`.`api_keys`.`created_at`, `main`.`api_keys`.`last_used_at`, `main`.`api_keys`.`expires_at`, `main`.`api_keys`.`user_id` FROM `main`.`api_keys` WHERE `main`.`api_keys`.`id` = ? LIMIT ? OFFSET ?, params: [2,1,0], result: "success", item_type: "query", is_query: true, duration_ms: 0
stump  |     at /usr/local/cargo/git/checkouts/quaint-453f07acedd929b5/c502995/src/connector/metrics.rs:87
stump  |     in quaint::connector::metrics::quaint:query with db.statement: SELECT `main`.`api_keys`.`id`, `main`.`api_keys`.`name`, `main`.`api_keys`.`short_token`, `main`.`api_keys`.`long_token_hash`, `main`.`api_keys`.`permissions`, `main`.`api_keys`.`created_at`, `main`.`api_keys`.`last_used_at`, `main`.`api_keys`.`expires_at`, `main`.`api_keys`.`user_id` FROM `main`.`api_keys` WHERE `main`.`api_keys`.`id` = ? LIMIT ? OFFSET ?
stump  |     in tower_http::trace::make_span::request with method: PUT, uri: /koreader/stump_REDACTED/syncs/progress, version: HTTP/1.1
stump  | 
stump  |   2024-11-16T09:36:44.875506Z DEBUG quaint::connector::metrics: query: COMMIT, params: [], result: "success", item_type: "query", is_query: true, duration_ms: 3
stump  |     at /usr/local/cargo/git/checkouts/quaint-453f07acedd929b5/c502995/src/connector/metrics.rs:87
stump  |     in quaint::connector::metrics::quaint:query with db.statement: COMMIT
stump  |     in tower_http::trace::make_span::request with method: PUT, uri: /koreader/stump_REDACTED/syncs/progress, version: HTTP/1.1
stump  | 
stump  |   2024-11-16T09:36:44.875841Z DEBUG quaint::connector::metrics: query: SELECT `main`.`media`.`id`, `main`.`media`.`name`, `main`.`media`.`size`, `main`.`media`.`extension`, `main`.`media`.`pages`, `main`.`media`.`updated_at`, `main`.`media`.`created_at`, `main`.`media`.`modified_at`, `main`.`media`.`deleted_at`, `main`.`media`.`hash`, `main`.`media`.`koreader_hash`, `main`.`media`.`path`, `main`.`media`.`status`, `main`.`media`.`series_id` FROM `main`.`media` WHERE `main`.`media`.`koreader_hash` = ? LIMIT ? OFFSET ?, params: ["ba9fd24a052ab00c717cc190eaf9ed5a",1,0], result: "success", item_type: "query", is_query: true, duration_ms: 0
stump  |     at /usr/local/cargo/git/checkouts/quaint-453f07acedd929b5/c502995/src/connector/metrics.rs:87
stump  |     in quaint::connector::metrics::quaint:query with db.statement: SELECT `main`.`media`.`id`, `main`.`media`.`name`, `main`.`media`.`size`, `main`.`media`.`extension`, `main`.`media`.`pages`, `main`.`media`.`updated_at`, `main`.`media`.`created_at`, `main`.`media`.`modified_at`, `main`.`media`.`deleted_at`, `main`.`media`.`hash`, `main`.`media`.`koreader_hash`, `main`.`media`.`path`, `main`.`media`.`status`, `main`.`media`.`series_id` FROM `main`.`media` WHERE `main`.`media`.`koreader_hash` = ? LIMIT ? OFFSET ?
stump  |     in tower_http::trace::make_span::request with method: PUT, uri: /koreader/stump_REDACTED/syncs/progress, version: HTTP/1.1
stump  | 
stump  |   2024-11-16T09:36:44.875902Z DEBUG tower_http::trace::on_response: finished processing request, latency: 5 ms, status: 404
stump  |     at /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/tower-http-0.5.2/src/trace/on_response.rs:114
stump  |     in tower_http::trace::make_span::request with method: PUT, uri: /koreader/stump_REDACTED/syncs/progress, version: HTTP/1.1

EDIT: I didn't read your answer properly. I did this with existing libraries so this might be the reason. I'll try recreating my libraries and report back.

[LnLcFlx]

I recreated my libraries but koreader_hash is still NULL

[aaronleopold]

Using STUMP_VERBOSITY=3 I get a 404 for /:api-key/syncs/progress in the logs

This makes sense. The hash doesn't exist in the database, so when koreader syncs the progress it won't find the book with the corresponding hash.

I recreated my libraries but koreader_hash is still NULL

Okay, so let's try to take inventory and see what we can sus out with a couple of hopefully helpful questions:

• The book you are trying to sync, what format is it in? Currently, only PDF and EPUB are having the hash generated. There are a few TODO comments in the code like this which ask whether other formats, like zip, should have the hash generated. I didn't have anyone offer to test this feature before initial merge, so I didn't implement it.

• When you scan a (fresh, new and/or empty) library with the koreader hashes enabled, are you able to search for the following keywords in the logs:

  Generated hash or generate_koreader_hash

  If so, please share some logs around them for hopefully helpful context

[LnLcFlx]

• The books are PDF or EPUB.
• Grepping my logs of my still running container I got matches so reviewed the SQLite database again and found the KOReader hashes to suddenly be there.
  Might it be, that stump does not hash them immediately after scanning or that it takes a long while?

I don't have my Tolino with me rn so I can't test it, but I strongly suspect that this is now fixed.

[aaronleopold]

Might it be, that stump does not hash them immediately after scanning or that it takes a long while?

No, if hashing is enabled it does it immediately as it constructs each book from disk.

[aaronleopold]

I don't have my Tolino with me rn so I can't test it, but I strongly suspect that this is now fixed

Once you have a moment to test it, please report back and I can close this out

[LnLcFlx]

It seems to work now.
However when I push my progress from koreader of a book that I haven't opened in stump yet, then the progress percentage is not being shown.

Also when pressing continue reading in stump, on a book whose progress is accurately reported as the progress pushed from koreader, it does not open the built-in reader on the proper page but on the last page opened in stump (for EPUB).

Also: Are you aware of way to sync my entire progress in koreader to stump all at once?

[aaronleopold]

However when I push my progress from koreader of a book that I haven't opened in stump yet, then the progress percentage is not being shown.

This is not expected, I can try to take a look this week. I'll leave this ticket open in the meantime.

Also when pressing continue reading in stump, on a book whose progress is accurately reported as the progress pushed from koreader, it does not open the built-in reader on the proper page but on the last page opened in stump (for EPUB).

This is a known limitation, perhaps it should be documented better. It should hopefully be improved down the road. For more context, it was discussed in the original issue starting around here. TLDR; KoReader uses x-pointer and Stump uses epubcfi, so a translation layer to/from is required.

Are you aware of way to sync my entire progress in koreader to stump all at once?

Unfortunately not, but if you find anything please share here and I can make a note in the docs! 🙂