Replies: 7 comments 22 replies
-
|
Thanks for the feedback. On a NASA you must use OSXPhotos is not optimized for NAS. There is an open issue to have NAS auto detected but this is low priority. OSXPhotos is also not optimized for Optimize Mac Storage. You could probably speed this up with I'm sorry it never completed for you. I've successfully exported larger libraries than yours with no problem. But I only use fast locally and attached storage. |
Beta Was this translation helpful? Give feedback.
-
|
Thanks for responding. I added the flags |
Beta Was this translation helpful? Give feedback.
-
|
Using --checkpoint 100 means the 1.2GB file has to be written to disk every 100 images. You're probably better off with a bigger number. |
Beta Was this translation helpful? Give feedback.
-
|
I've been using osxphotos on a 100K pics Library into NAS Synology successfully for a few year already 😄 ! I also use --cleanup to delete any pics on the exported folder that I happen to delete from Photos. I don't use --photo-kit. I don't use iCloud storage or the need for downloading images from iCloud due to the "Optimize Mac Storage"... which I believe can slow things down. First backup can take time.. especially if you have videos. I have 1 GBPs network and my exports can take on first export,or when a large number of files need to be updated, 8~10 hours for ~100K pics. but subsequent daily's with a few thousands pics changed requiring update/copy sits around 4 hours. From these somehow outdated tests (#582 (comment)) on 1GB LAN it was taking about 0.3 seconds per pic (not huge Videos!) which would yield ~16.6 hours for 200K pics. If you incorporate --verbose and --timestamp options you get an output like the following and track progress and performance. In the example below it tells you:
(...)
2024-10-21 16:27:25.604177 -- Exporting XXX.HEIC (XXX.HEIC) (89000/107186)
2024-10-21 16:27:25.697627 -- Skipped up to date file /Volumes/photo/XXX.HEIC
2024-10-21 16:27:25.705689 -- Exporting ZZZ.JPG (ZZZ.JPG) (89001/107186)
2024-10-21 16:27:25.807583 -- Skipped up to date file /Volumes/photo/ZZZJPG
(...)
2024-10-21 16:27:11.212614 -- Exporting IMG_WRITTEN.jpg (IMG_WRITTEN.jpg) (107082/107186)
2024-10-21 16:27:11.327217 -- Writing metadata with exiftool for IMG_WRITTEN.jpg
2024-10-21 16:27:11.488014 -- Exported IMG_WRITTEN.jpg.jpg to
(...)
2024-10-21 16:27:26.256606 -- Elapsed time: 4:36:09
(...)
2024-10-21 16:27:26.303988 -- Cleaning up /Volumes/photo
2024-10-21 16:31:36.375673 -- Deleting /Volumes/photo/TTTjpeg
2024-10-21 16:32:59.701735 -- Deleted: 1 file, 0 directories |
Beta Was this translation helpful? Give feedback.
-
|
@oPromessa Thanks for the tip on the To be clear, I fully expected the first complete backup to take many days to complete, but with every export run starting back at zero and never getting past more than ~20k pictures in a day, it seemed like it was never going to finish. I understand that "Optimize Storage" requires downloads from iCloud to get missing data (which I imagine significantly slows down the export), but I can't fit my entire library on my MacBook. However, if you can fit your entire photo library on a local drive why not just backup the Photos Library directly instead of using OSXPhotos? |
Beta Was this translation helpful? Give feedback.
-
|
Further to this I have a small python script which detect lengthy operations (over 60secs for instance) from a --verbose/--timestamp osxphotos log file: I run For my case, you can see in some cases a exiftool+copy of 214 seconds... this was for a 1.3 GB MP4 videos Sequence: 1 Line: 234261 over [ 60 secs] 75.45 Duration: 00:01:15.449956 2024-10-20 04:15:01.670564 -- Writing metadata with exiftool for XXX.MOV
Sequence: 1 Line: 252845 over [ 60 secs] 68.53 Duration: 00:01:08.525544 2024-10-20 04:49:53.517625 -- Writing metadata with exiftool for XXX.MP4
Sequence: 1 Line: 274033 over [ 60 secs] 214.69 Duration: 00:03:34.686064 2024-10-20 05:29:36.144517 -- Writing metadata with exiftool for XXX.MP4
Sequence: 1 Line: 282734 over [ 60 secs] 69.05 Duration: 00:01:09.050918 2024-10-20 05:47:40.673230 -- Writing metadata with exiftool for XXX.mp4If one compares with rsync for the same file and origin/destination.. it's much faster. But then again it is not performing several databases and comparisons operations required and also applying/using exiftool changes! |
Beta Was this translation helpful? Give feedback.
-
Strange.. it should not start from zero! You can try the --limit option to incrementally backup. osxphotos would then pick/up from where it left-off -- I still think it must check no changes occurred on previously exported files but if would be faster. @RhetTbull doesn't like it -- joking 😄 -- but I also use --exportdb and keep the database local. Avoids a few seconds locking it up into memory and writing it back to disk at the end. But during the copy should not make a difference. Look at --retry option to accommodate for transient copy over the LAN errors. Also note the --cleanup option should you choose to use it deletes all unknown files. Use the --keep if you have any files you want to keep on the export folder which are not Pics/Movies related.
In my case I export into Synology Photos and is not only for backup purposes. With Synology Photos I can make all the pics replicating the Photos structure and nicely organised in Albums, tagged with keywords (except videos), geolocation, etc.. and make it available via the Web (behind CloudFlare) and via Mobile app Synology Photos Mobile for the whole family to access on-the-go anytime anywhere. |
Beta Was this translation helpful? Give feedback.
-
You're missing the point of OSXPhotos. It's designed to export photos and metadata so you have an archive copy outside the Apple ecosystem. Backing up the Photos library still keeps your photos locked in a proprietary database. OSXPhotos also gives you full control over the output. If what you really want to do is just backup from iCloud then iCloud Photos Downloader might be a better option. |
Beta Was this translation helpful? Give feedback.
-
|
@RhetTbull and @oPromessa thank you again for your suggestions and feedback. I let OSXPhotos run overnight with the following command line: Which backed up 11987 files in 11 hours and 28 minutes which is about 17 seconds per file. I'm going to start a second export run this morning (changing to
When I said "start from zero" I mean that every export always starts with a line like: So it's not possible to see how much of the library has already been exported by previous runs. I thought that each subsequent run would be slightly faster as it should be skipping files that have already been exported--and I can see messages in the log that it is skipping files--but the speed of the export seems to be roughly the same even with skipped files meaning that it never reaches the end. Thanks for the tip about iCloud Photos Downloader. I had already experimented with it before discovering OSXPhotos. I was uncomfortable with the stability of their approach which is basically screen-scrapping iCloud.com and concerned about the potential for getting my account locked for using it. (Probably being paranoid, but it's certainly not an "Apple approved" method for downloading your photos). I think OSXPhotos has a more robust architecture and uses the official PhotoKIt API (or AppleScript) for downloading photos. Also, OSXPhotos is the only tool I'm aware of that has an import feature that allows importing an exported archive of photos back into a Photo library. |
Beta Was this translation helpful? Give feedback.
-
|
Yup 17 seconds per file whereas I have 0.3 seconds is quite a big difference. This is a bit beyond the scope of osxphotos -- as Rhet mentioned Again, unless your files are quite big Videos... You may have to check networking... there are info out there on optimising LAN to NAS... and also smoother optimisations.
|
Beta Was this translation helpful? Give feedback.
-
|
You could try using The |
Beta Was this translation helpful? Give feedback.
-
|
See also #1086 |
Beta Was this translation helpful? Give feedback.
-
|
My second export run this morning averaged ~3 seconds per file, which was what I was expecting after spot-checking the log as it was running. Perhaps my previous overnight run got stuck on something or the laptop went to sleep? Anyway, I'm trying a third run with In the meantime, I think the conversation has veered off into the performance details whereas as my problem is more with the progress reporting and/or lack of forward progress on the export. I fully expected the first backup to take a long time and TBH performance is not critical and I don't really mind if it takes 10 days or 40 days to complete The problem is that progress reporting doesn't tell me how much of the backup has been completed and it seems like there is no forward progress. For example, I expected that exports were "resumable" and after the first partial export, subsequent exports would report something like "100,000 files already exported, exporting remaining 200,000 files". Instead subsequent exports start again from "zero" and skip over the already exported files, but skipping over the exported files doesn't seem any faster then exporting them in the first place, and it's difficult to tell if we are making any forward progress on the full backup. My use case is the same as @torarnv in this comment #1086 (comment) From reading through that thread it seems like this is a use case the OSXPhotos doesn't support well? |
Beta Was this translation helpful? Give feedback.
-
Exports are resumable. But OSXPhotos still needs to check each photo to see if something has changed between Photos and the exported files. For example, you edited a photo or added a keyword. My priority is maximum preservation of data, not speed. To do this OSXPhotos must look at the exported file to see if it's different than expected. Over a slow network/NAS this takes time. OSXPhotos is not optimized for this use case. You can use There are inefficiencies in the design that are particularly bad for your use case (optimized library + NAS). The previously mentioned issue is to track this and I'll eventually get around to looking at it but it's not a priority as it doesn't affect my daily use of OSXPhotos. (And I only have a few hours per week to work on this). I'm happy to consider PRs or even suggestions to improve the checks in the code. As for never finishing, I don't know what the issue is but if there is a bug I'd definitely like to squash it. Unfortunately without being able to replicate it it's hard to know what the issue is. If OSXPhotos crashes it creates a crash log with debug info but it sounds like in your case it's just hanging. |
Beta Was this translation helpful? Give feedback.
-
|
There are some very very odd networking issues going on with Sequoia right now. I've experienced two, one of which I'll describe at the end for the curious. I've also seen many reports of users having issues with NASs and Sequoia. Googling "Sequoia Networking" or "Sequoia networking issues" will show plenty of issues. Not saying that this has anything to do with what the OP is running into, but it bears at least some consideration. Definitely impacting NAS usage as well. The more interesting issue I had (still have!) is iMessage not working with VPN on one Mac, and only this one Mac. Same OS and VPN (and version) on all other Macs, no issue, but for this Mac, iMessage just doesn't work. For a while it worked when I turned the VPN off, but then it stopped working at that point too. I escalated to Apple at that point and my call was escalated within Apple, it was a 2+ hour data collection fun-fest. They know and acknowledge this is a bug, but as of 2 weeks ago, they were still very confused as to what was happening. Folks are pointing to the firewall, but during the call we turned it on and off and lots of other combinations and it had no effect. While yes this is an Internet issue, I've run into odd LAN issues as well that are hard to quantify. 15.0.1 did not fix any of the issues I have or have seen. |
Beta Was this translation helpful? Give feedback.
-
|
@cfc62 that's why I'm not on Sequoia yet and why OSXPhotos isn't fully tested on Sequoia. I prefer to stay one version behind. |
Beta Was this translation helpful? Give feedback.
-
|
Here's the end of the log of my last run with the profiling data. I am not using Sequoia. The
I can see how this would hurt performance especially when connecting to a NAS. If you made the assumption that OSXPhotos was the only writer to the export directory and no modifications were being made to files by the user you could you rely entirely on the exportedb without checking the photo files to see if it has changed?
To be clear I didn't meant that the backup hung or crashed, rather that I was never able to run a en export to completion after multiple runs over many days. Due to the way progress is reported it was difficult to tell how much of the library has already been exported and it seems that that on a NAS resuming an export is not significantly faster than the first export. It sounds like |
Beta Was this translation helpful? Give feedback.
-
Doesn't the For future reference and anyone else following along, here are the final command-line switches I settled on: |
Beta Was this translation helpful? Give feedback.
-
|
|
Beta Was this translation helpful? Give feedback.
-
|
That would be great. Just wondering the delete operation to "cleanup files previously exported which were in the meanwhile deleted from Photos" would continue to work/applied, correct? |
Beta Was this translation helpful? Give feedback.
-
|
|
Beta Was this translation helpful? Give feedback.
-
Beta Was this translation helpful? Give feedback.
-
|
@gerdemb Not sure if this will be helpful at all as it seems you already have optimized speed / connection between your NAS and Mac. If you haven't you may want to see some of the info in these two posts: Post 1 and Post 2 especially if you are using SMBv1. |
Beta Was this translation helpful? Give feedback.
-
|
@rajscode thanks for the suggestions. I am sure that are some things I could do to try and speed up the data transfer between my Mac and NAS, but I run some other jobs with large data transfers and none of them run nearly as slowly as |
Beta Was this translation helpful? Give feedback.
-
|
Efficient vs. Large Transfers. Efficient file transfer with osxphotos isn’t just about total data size. For example, transferring 500 files totaling 2TB may work smoothly, but transferring 200,000+ smaller files can become inefficient due to configuration issues like:
Even with a local SSD storing an optimized iCloud Photos library, exports via osxphotos are inherently slow because each run downloads every photo from iCloud.com. My Setup for a Large Library (160K+ Photos, 6.5K Videos, 1.7TB) To maintain a full-resolution backup outside the Photos app:
This structure lets me manage my media in a standard directory format, enabling easy backups to a NAS or cloud storage. For details, see posts on MacRumors: (High Level Backup Stragegy and Details of Dual Library on a single Mac) EDIT ADD:
|
Beta Was this translation helpful? Give feedback.
-
This is true. OSXPhotos will stage each file to a temporary directory before it processes it. This vastly simplifies the internal logic and if on a local APFS disk, there's no performance penalty due to copy-on-write. If Photo library is on a different disk or across the network, this slows things down a lot. If the file has not been downloaded from iCloud it will also have to be downloaded then staged which further slows things down. OSXPhotos is not yet optimized to work with "optimized" libraries though I'm slowly making progress to improve this. See #1086. Secondly, OSXPhotos does a "stat" on the target directory for every single file that is copied. Over a network this can be very slow, especially if there are lots of files in the target directory. The reason for this is to ensure the desired target name is available (doesn't already exist on the target volume) and to see if the target copy differs from expected and needs to be updated (this part can be skipped with |
Beta Was this translation helpful? Give feedback.
-
|
@gerdemb Completely outside of Photos/Osxphotos I do keep track of other assets and absolutely there can be (and are) very very large disparities in performance networked vs. direct attached storage. As an example if I open up this other database with the assets available over the network, time from start to a working session can be 4-5 minutes and more, where if I open up the database with the assets directly attached, it takes 15-20 seconds at most. That's perhaps 20 to 30x faster on direct attached storage. It's not related network load in any way, neither the network accesses nor the disk accesses are anything substantial at all. PS: the network is fully wired, no wi-fi inconsistencies involved. It's 1G, based on what I've seen I doubt going to either 2.5G or 10G would make much of a difference. |
Beta Was this translation helpful? Give feedback.
-
Hello!
I've been testing osxphotos to back up my photo library to my NAS, and I wanted to share some feedback and my experiences with the tool. My use case is to back up my entire library to my NAS. I have 267,296 items in my library that take up 1.3TB of storage in iCloud with "Optimize Storage" enabled.
First off, I've been looking for a tool like this for a long time and I want to acknowledge the incredible effort put in by the community to make osxphotos as powerful and flexible as it is. I want to emphasize that my comments are intended as constructive criticism, to help make osxphotos even better.
Initially, I found the number of command-line options a bit overwhelming as a beginner. I ended up reading all the documentation before I felt confident I had the correct combination of switches to do a full backup; however, I still wasn't entirely sure I had the right combination and it seems like the that combination could be a moving target as new versions are released. I think osxphotos would benefit from an option similar to rsync's
--archive, which provides a sensible default combination of switches. The use case of "export my entire library while preserving as much information as possible" seems pretty standard. For reference, here's the combination of switches I settled on:Related GitHub discussions:
#1267
#1659
#1515
Another issue I encountered was with performance and progress reporting. I ran this command continuously for several hours every day for several weeks, but it never seemed to come close to completing a full export. At the beginning of each run, the export always started from zero, making it unclear how much had been completed in previous runs. On each subsequent run, I could see more "skipped up-to-date file" messages, but it seemed to take just as much time to "skip" a file as it did to back up a new one. In the end, the highest number of completed files exported was around 20,000, or about 10% of my library, despite running for 6-8 hours per day for a couple weeks. Eventually, I gave up, as it was never able to complete a full backup. I realize my library is probably larger than average, which could have contributed to the performance issues. Or perhaps I had some setting wrong?
Finally, (and I realize this is a small nitpick), CTRL-C wouldn't stop the export on my machine, and I always had to force-kill the process.
Thanks again to the developers and community for all the effort that has gone into making osxphotos such a useful tool. I hope my feedback can help improve it further!
Cheers,
Ben
As an aside, after failing to complete a full backup with osxphotos, I created a simple photo backup app inspired by Time Machine. It uses the Apple PhotoKit API, so it lacks the advanced features that osxphotos has by reverse-engineering the Photos database, but it offers much faster performance for my large library. If you're curious, there's a link in my GitHub profile.
Beta Was this translation helpful? Give feedback.
All reactions