Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

implement partial library scan (filescanner) #1179

Open
wants to merge 6 commits into
base: master
Choose a base branch
from

Conversation

whatdoineed2do
Copy link
Contributor

Fixes #382, workaround to #1173

Extend library interface to provide a rescan path entry point and expose on json api. Accepts a path for scan - currently implemented in filescanner and allows for partial local library (re)scanning.

Could be future extended for RSS lib and spotify lib scanners, with RSS potentially forcing refresh of items under a server (ie a path being https://bbc.co.uk/rss/ to force rescan of these hosted RSS).

Uses existing filescanner.c's bulk_scan() function but extended to take a requested path otherwise use library directories as existing functionality.

@chme
Copy link
Collaborator

chme commented Jan 23, 2021

The last time I looked into implementing this, I ran into problems with the purge_cruft(starttime) deleting to many items. I may have missed something, but I don't see how you make sure, that only items that are under the rescan path are deleted. Additionally I remember symlinks to be a potential problem.

@whatdoineed2do
Copy link
Contributor Author

Thanks for that - I was so tunnel visioned on the artwork piece i missed that rather large problem!

I've given this a wider stress test today and it appears to resolve the problem on deleting too many items by doing a negative ping on files, playlists and the directories (ie all items that are not the path being scanned are marked as good) and also the inotify table is not cleared down but only the path in question being cleared out.

@whatdoineed2do
Copy link
Contributor Author

@ejurgensen @chme
Is there any comments for this PR?

I've had this on my own branch for a while and its been fine - I use it most on a slow (secondary) RPi 1 that has its (largish) media library on a remote NFS and configured not to rescan on startup - changes on NFS arent picked up by inotify and thus when I modify existing albums I can force a rescan of that specific path rather than force full libraary rescan.

@ejurgensen
Copy link
Member

Sorry haven't looked at it. I will try to get around it.

@whatdoineed2do
Copy link
Contributor Author

@chme - i rebased this against the recent changes in master for the partial scanning basd on scna type. with that work fresh in your memory are you able to give any comments on this PR?

@whatdoineed2do
Copy link
Contributor Author

@ejurgensen - any thoughts on this PR?

@ejurgensen
Copy link
Member

I'm a bit unsure about the intended use of this, could you share a bit on that? I mean, I get that it means that you can trigger a partial rescan, but how is the user supposed to use it? Is it the goal that advanced users can integrate in library update scripts via the api, or is it the goal that users in the web UI can navigate to the directory they want rescanned? Both seem a bit clunky to me, but maybe there isn't a better alternative.

@whatdoineed2do
Copy link
Contributor Author

In my own branch, I've built the UI extensions to trigger rescan which i've kept out of this PR
image

But the use case is the latter you described and echo'd on #382 - large library mounted on NFS, I add a new album under the artist directory like above (/media/music is on NFS), because its NFS, inotify doesnt pick up the files dropped onto the remote NFS server and thus doesn't appear on the owntown server using NFS. With this, patch, I'd trigger a rescan on the artist directory which will find the new album i've added instead of scanning the full remote NFS library

This is a little pain for my ancient RPI1 - scanning my ~10k files takes about 2.5 mins rather than the few seconds it would for scanning the specific directory.

@ejurgensen
Copy link
Member

Thanks for the elaboration, I will try to get around to looking at the change.

Just out of curiousity, have you tried something like this https://github.com/mhallin/notify-forwarder? That particular project seems like it is dead, but that way of solving the problem might have some merit.

@benediktkr
Copy link

i have successfully rebased the file-scan-dir-path from your whatdoineed2do/forked-daapd repo, rebased it off the current HEAD of he main branch first. works great and i can use the api to trigger rescans for a given path, thank you :)

In my own branch, I've built the UI extensions to trigger rescan which i've kept out of this PR

which branch is that? would be great to have it in the UI like i can see in your screenshot.

@whatdoineed2do
Copy link
Contributor Author

@benediktkr - thanks for testing. It should work as intended; i've been using it since the original submission although there were concerns raised by others about accuracy of the rescans impacting existing items in db (i've not observed though)

In my own branch, I've built the UI extensions to trigger rescan which i've kept out of this PR

which branch is that? would be great to have it in the UI like i can see in your screenshot.

This might be messy as my tree (my master etc) have fixes that work for me but not been accepted into the upstream, that incls web work. If you can do a bit of manual cherry-picking and to rebuild the UI I can point you at the files of interest:

@benediktkr
Copy link

benediktkr commented Jun 23, 2023

I actually like the web ui changes youve made :) so im fine with using them.

slightly related, but ive been trying to get a dark theme working, using a css file that the Dark Reader addon for Firefox has generated. I havent gotten it to work though (but it works great in my browser using the addon itself).

if youre interested you can find it in my own builds of OwnTone at git.sudo.is/ben/builds/owntone.

@Bockiii
Copy link

Bockiii commented Jul 8, 2024

In case you are still following this up @whatdoineed2do or @benediktkr : I would very much love something like this, even though my use case is different.

I have the issue that my downloader seems to still have "a finger" on the file when it puts it into the download folder (if I understand the code correctly it downloads the content into the mp3 and then updates the id3 tags etc, but by that time, the folder watcher has already retrieved the non-existing info). So I have to do a "rescan all with metadata" every time I add something, because it wont update the metadata for just that one album or so. My collection is 100k+ files so its getting a little stupid in regards to runtime and also file reads (nothing to wear out a drive than to metadata scan 100k files for no good reason once in a while).

So if you could get this into owntone, it would be awesome.

As for the usecase: I would argue that most people that run owntone are advanced users in some manner, so that adding an additional scan option like this wont make a difference.

Plex lets you do it too ;)

@ejurgensen
Copy link
Member

@Bockiii that sounds like something isn't working as it should. Would you mind opening an issue with debug logging of when the downloader adds a track?

Btw if the downloader is a script you might be able to add a 'touch' command to avoid a full rescan.

@Bockiii
Copy link

Bockiii commented Jul 8, 2024

The downloader is deemix and its development has been abandoned a while ago (but its still working until they fix the loophole). So I cant open a PR with the deemix dev :)

I will open another issue for this

@whatdoineed2do
Copy link
Contributor Author

In case you are still following this up @whatdoineed2do or @benediktkr : I would very much love something like this, even though my use case is different.

I have the issue that my downloader seems to still have "a finger" on the file when it puts it into the download folder (if I understand the code correctly it downloads the content into the mp3 and then updates the id3 tags etc, but by that time, the folder watcher has already retrieved the non-existing info). So I have to do a "rescan all with metadata" every time I add something, because it wont update the metadata for just that one album or so. My collection is 100k+ files so its getting a little stupid in regards to runtime and also file reads (nothing to wear out a drive than to metadata scan 100k files for no good reason once in a while).

So if you could get this into owntone, it would be awesome.

As for the usecase: I would argue that most people that run owntone are advanced users in some manner, so that adding an additional scan option like this wont make a difference.

Plex lets you do it too ;)

@Bockiii
I've been using this feature in my own tree for a number of years now and I think its ready (its never destroyed my DB :) but this is awaiting PR feedback - I don't own the merge decision

However, its less of a push for me since, as I say, I have my own tree with own features I wanted that were not accepted into the upstream.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

New feature: partial lib re-scan
5 participants