Replies: 13 comments 38 replies
-
|
Try using --ramdb |
Beta Was this translation helpful? Give feedback.
-
|
Also, note that the label (computed by Photos) can change as Photos does more analysis on the library which can cause the metadata to be updated with |
Beta Was this translation helpful? Give feedback.
-
|
You might also want to store the export database on your local Mac using --exportdb Edit: if you do this you'll want to move the export database to the new location. |
Beta Was this translation helpful? Give feedback.
-
|
I update from MAC with SSD to a Synology NAS (no SSD) via 1Gbit with pretty much your parameters plus the ones referenced by RheTbull:
And for a ~90K Photos database it takes about 4 hours with smaller updates and around 5 hours if I have say ~200 new pics and videos. The wildcard is when Photos makes updates on things like labels (as mentioned) as it may be updating PS: I had a side issue which osxphotos would always export pics dats before 1970-01-01 as the NAS Synology would not record the date/time correctly... since then solved on the NAS by the newer versions. My logging times on my last runs... 6374275 [notice] --- Start Sun 28 May 2023 03:44:01 WEST p= 1 LONG=
2023-05-28 03:44:04.420476 -- osxphotos version: 0.59.3
2023-05-28 03:45:56.378065 -- Exporting 92661 photos to /Volumes/photo/XXX
2023-05-28 07:39:53.910683 -- Processed: 92661 photos, exported: 2007, updated: 313, skipped: 90343, updated EXIF data: 2320, missing: 0, error: 0, touched date: 326
2023-05-28 07:39:53.911315 -- Elapsed time: 3:53:57
2023-05-28 07:52:40.759952 -- Deleted: 10 files, 1 directory
[notice] Result: 0
6569002 [notice] --- End Sun 28 May 2023 07:54:13 WEST p.end= 0
6569003 [notice] --- Start Mon 29 May 2023 03:44:00 WEST p= 1 LONG=
2023-05-29 03:44:03.663441 -- osxphotos version: 0.59.3
2023-05-29 03:45:22.014169 -- Exporting 90651 photos to /Volumes/photo/XXX
2023-05-29 07:30:26.912168 -- Processed: 90651 photos, exported: 23, updated: 1852, skipped: 88778, updated EXIF data: 1875, missing: 0, error: 0, touched date: 953
2023-05-29 07:30:26.912802 -- Elapsed time: 3:45:05
2023-05-29 07:39:52.700939 -- Deleted: 2033 files, 0 directories
[notice] Result: 0
6759435 [notice] --- End Mon 29 May 2023 07:41:08 WEST p.end= 0
6759436 [notice] --- Start Sat 3 Jun 2023 03:44:00 WEST p= 1 LONG=
2023-06-03 03:44:04.930036 -- osxphotos version: 0.60.1
2023-06-03 03:45:26.032820 -- Exporting 90651 photos to /Volumes/photo/XXX
2023-06-03 08:05:59.588303 -- Processed: 90651 photos, exported: 5, updated: 710, skipped: 89938, updated EXIF data: 715, missing: 0, error: 0, touched date: 220
2023-06-03 08:05:59.589010 -- Elapsed time: 4:20:33
2023-06-03 08:15:49.050728 -- Deleted: 5 files, 0 directories
[notice] Result: 0
6944785 [notice] --- End Sat 3 Jun 2023 08:21:44 WEST p.end= 0
6944786 [notice] --- Start Sun 4 Jun 2023 03:44:00 WEST p= 1 LONG=
2023-06-04 03:44:05.201415 -- osxphotos version: 0.60.1
2023-06-04 03:45:31.476632 -- Exporting 90778 photos to /Volumes/photo/XXX
2023-06-04 07:27:23.836433 -- Processed: 90778 photos, exported: 127, updated: 92, skipped: 90561, updated EXIF data: 219, missing: 0, error: 0, touched date: 55
2023-06-04 07:27:23.837147 -- Elapsed time: 3:41:52
2023-06-04 07:37:39.049411 -- Deleted: 0 files, 0 directories
[notice] Result: 0
7129271 [notice] --- End Sun 4 Jun 2023 07:39:19 WEST p.end= 0
7129272 [notice] --- Start Sat 10 Jun 2023 03:44:00 WEST p= 1 LONG=
2023-06-10 03:44:16.600844 -- osxphotos version: 0.60.1
2023-06-10 03:45:12.573836 -- Exporting 90954 photos to /Volumes/photo/XXX
2023-06-10 08:53:03.976388 -- Processed: 90954 photos, exported: 430, updated: 1202, skipped: 89324, updated EXIF data: 1632, missing: 0, error: 0, touched date: 359
2023-06-10 08:53:03.977041 -- Elapsed time: 5:07:51
2023-06-10 09:00:59.446219 -- Deleted: 254 files, 0 directories
[notice] Result: 0
7317496 [notice] --- End Sat 10 Jun 2023 09:02:29 WEST p.end= 0 |
Beta Was this translation helpful? Give feedback.
-
|
This comes out to about 6 photos per second which still seems awfully slow. If you run with --profile (which must be passed before export, e.g. |
Beta Was this translation helpful? Give feedback.
-
|
I tried --ramdb. Also tried locating the temporary folder on the same drive as the photos (external APFS). Neither seemed to have material impact on the speed. Now running --update with --profile to see what happens. 3 more hours to go. I did --verbose and it seems the time is spent on this: Exporting IMG_4655.HEIC (BC9325FA-0468-4D37-83F9-9C365DE65306.heic) (2/111314) It only does 2-5 of those per second, which is ok-ish speed for export but seems too slow for just checking if a file is updated or not. Could it be that the Photos database itself gets slow due to 110K photos in it? |
Beta Was this translation helpful? Give feedback.
-
Doubtful, it's probably an osxphotos issue. Osxphotos loads the photos database at startup then operates from an in-memory copy. It could be the in-memory version is so large it's causing a lot of memory swapping. Look in Activity Monitor and see how much memory python is using. |
Beta Was this translation helpful? Give feedback.
-
|
Python is using 13GB right now, which seems not too much for a 32GB machine with nothing else major running. |
Beta Was this translation helpful? Give feedback.
-
|
This is what I got after running an update with a --profile: Exporting 111314 photos to /Volumes/photos/icloud export... Ordered by: cumulative time ncalls tottime percall cumtime percall filename:lineno(function) |
Beta Was this translation helpful? Give feedback.
-
|
Thanks for the profile data. A lot of time spent in AppleScript ( |
Beta Was this translation helpful? Give feedback.
-
|
That's somewhat strage because --download-missing should do nothing - all images were downloaded when it was run (it says 55 missing but the library is entirely locally stored). And --exiftool should do almost nothing because only a handful of files out of 110K had any metadata changes. So, in ideal scenario, --download-missing should have had no effect because there is nothing missing and --exiftool should have negligible impact because it's relevant to just 17 files. I will try dropping --download-missing - it's there only "just in case" in my setup. |
Beta Was this translation helpful? Give feedback.
-
|
By the way, massive thanks for making this awesome tool! It was part of the reason for me moving to Apple Photos (from Lightroom CC). Bi-directional sync would be even better, but even without it, it's super-useful! |
Beta Was this translation helpful? Give feedback.
-
What do you mean by this? There is an |
Beta Was this translation helpful? Give feedback.
-
re: Permission if doing the above as a scheduled job the following may be of some help: Privacy & Security sections: Files and Folders:
Full Disk Access:
Photos:
Further details at this discussion item |
Beta Was this translation helpful? Give feedback.
-
I've done some debugging on my end and it appears the "download missing" path is being called even if not required anytime the In the meantime, if you are using "keep originals on this Mac" you should have few if any missing photos so you should be able to run without |
Beta Was this translation helpful? Give feedback.
-
|
@MaxLyt Please update osxphotos to v0.60.2. I think you'll find the update speed is much improved as the "download missing" code path is no longer being called when files don't actually have to be downloaded. For my library, my update was taking 2 hrs 18 min on v0.60.1 and it now takes 0 hrs 47 min on v0.60.2 for 37,000 images. |
Beta Was this translation helpful? Give feedback.
-
|
Would it also improve exports of libraries with only local pics disconnected from iCloud and this without the need to use the |
Beta Was this translation helpful? Give feedback.
-
|
@oPromessa Only if you're using |
Beta Was this translation helpful? Give feedback.
-
|
MASSIVE SPEED improvement on the job I run: 'osxphotos export --download-missing --use-photokit --load-config' toml listed further below. v0.60.1: ~ 6 1/2 - 7hrs each run Details: toml contents: [export]
# Only export new or updated files.
update = true
cleanup = true
report = "/Volumes/WDSSD2TB/photo_library_export/osxphotos_report_backup.sqlite"
append = true
db = "/Volumes/WDSSD2TB/Macbook_Pro_14_PhotosLibrary_Full_Originals_Backup/Photos Library.photoslibrary"
tmpdir = "/Volumes/WDSSD2TB/photo_library_export/osxphotos_tmp"
directory = "{created.year}/{created.year}-{created.mm}"
theme = "dark"
sidecar = [ "xmp",]
sidecar_drop_ext = true
person_keyword = true
favorite_rating = true
keyword_template = [
"Album: {folder_album}",
"{label}",
"{favorite?is_favorite}",
"is_{media_type}",
] |
Beta Was this translation helpful? Give feedback.
-
|
Just tried - ~11 times improvement in speed for me. Quite amazing! |
Beta Was this translation helpful? Give feedback.
-
|
Hello -- Noticed big slow down on export. All had been good speed wise on improvements discussed above up to v0.64.2 which I'd been using until 5 days ago when I upgraded to v0.67.2. At that point elasped time for the export has now gone up 3x-4x longer. I've now run it 4 times and it is staying around the 5.5-6 hr duration now for the export (used to be ~ 1 1/2 hrs using v0.64.2 and earlier). Thanks for any guidance ! Last run results: > osxphotos -v
> osxphotos export /Volumes/WDSSD2TB/photo_library_export/photo_library_backup/ --ramdb --download-missing --use-photokit --load-config /Volumes/WDSSD2TB/photo_library_export/osxphotos_backup.tomlosxphotos, version 0.67.2
Python 3.12.1 (main, Dec 7 2023, 20:45:44) [Clang 15.0.0 (clang-1500.0.40.1)]
macOS 14.2.1, arm64
Exporting 210953 photos to /Volumes/WDSSD2TB/photo_library_export/photo_library_backup...
Exporting 210953 photos ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 100% 0:00:00
Processed: 210953 photos, exported: 30, updated: 70, skipped: 263269, updated EXIF data: 0, missing: 75, error: 0
Elapsed time: 5:27:49
Cleaning up /Volumes/WDSSD2TB/photo_library_export/photo_library_backup
Deleted: 1 file, 0 directoriesContents of config toml file used in export command above: /Volumes/WDSSD2TB/photo_library_export/osxphotos_backup.toml [export]
update = true
cleanup = true
report = "/Volumes/WDSSD2TB/photo_library_export/osxphotos_report_backup.sqlite"
append = true
db = "/Volumes/WDSSD2TB/Macbook_Pro_14_PhotosLibrary_Full_Originals_Backup/Photos Library.photoslibrary"
tmpdir = "/Volumes/WDSSD2TB/photo_library_export/osxphotos_tmp"
directory = "{created.year}/{created.year}-{created.mm}"
theme = "dark"
sidecar = [ "xmp",]
sidecar_drop_ext = true
person_keyword = true
favorite_rating = true
keyword_template = [
"Album: {folder_album}",
"{label}",
"{favorite?is_favorite}",
"is_{media_type}",
] |
Beta Was this translation helpful? Give feedback.
-
|
For now I've chosen to leave the |
Beta Was this translation helpful? Give feedback.
-
|
Findings: v0.67.3 is slower compared to v0.64.x BUT removing the checkpoint every 1000 did help reduce the time delta by ~ 1/2. v0.64.2 - Elasped time: 1:35:48 (export command) export command: |
Beta Was this translation helpful? Give feedback.
-
|
@rajscode Do you have the profile data from the run you ran with --profile? |
Beta Was this translation helpful? Give feedback.
-
|
Apologies for the delayed response. I'd killed off original run once it was apparent it was thrashing the drive. Just ran --profile command: Attached is the profile output |
Beta Was this translation helpful? Give feedback.
-
|
Thanks for the profile data. It's not surprising the profiled run took longer as turning on profiling slows things down quite a bit. The bits of coding that appear to be taking up the most time are: Listing the directory (to check for file collisions and verify if the file exists) took 10 minutes. Executing sqlite commands (which includes reading the Photos database and reading/writing the export database) took a cumulative 66 minutes. Finally, The biggest change between these versions of osxphotos is the addition of saving the history to the database so it's possible to know why a photo was skipped or updated. But it seems this added very little total time (a couple of seconds): |
Beta Was this translation helpful? Give feedback.
-
|
@MaxLyt I also export to NAS and did not notice this impact on the osxhotos version update as I use |
Beta Was this translation helpful? Give feedback.
-
|
@oPromessa You'll want to upgrade to v0.67.3 as earlier versions since v0.65.0 were causing unnecessary thrash to backup the export database during the export. In your case, this will cause unnecessary writing to the SSD which should be avoided. |
Beta Was this translation helpful? Give feedback.
-
Thanks. Is it possible to downgrade back to v0.64.2 and run the export with the profile turned on ? Kinda curious as to what might have caused the export to go up approx 73% in time to complete [ ~ 1hr35min on v64.2 vs 2hr44min on v67.3 ]. FYI - I do stop backup processes (Backblaze) while the export job is running so they don't impact the export job speed. Not a big deal though as it's a job that runs around 2am for me once a week. Thanks! |
Beta Was this translation helpful? Give feedback.
-
|
Let me run a test before you downgrade. In general osxphotos does not support downgrading due to schema changes in the export database. We might be able to hack it to make this work for a test. |
Beta Was this translation helpful? Give feedback.
-
|
@rajscode I ran a test and it looks like this will work. osxphotos the only change in export database schema is the addition of the history database and because the export database version is >= the version expected by 0.64.2, it runs fine without complaint. However, I strongly recommend you backup the export database (and the associated .wal and .shm file if they exist) before doing this. You can run osxphotos 0.64.2 without installing it using pipx:
|
Beta Was this translation helpful? Give feedback.
-
|
Trying to figure out a way to get 0.64.2 to install / execute even an osxphotos -v initially prior to trying the export. My machine has the following: pipx v 1.4.3 Any which way I've tried to to get v 0.64.2 run I get errors when running osxphotos -v. i.e. ❯ pipx run osxphotos==0.64.2 -v
⚠️ osxphotos is already on your PATH and installed at /Users/rajscode/.local/bin/osxphotos. Downloading and running anyway.
Fatal error from pip prevented installation. Full pip output in file:
/Users/rajscode/.local/pipx/logs/cmd_2024-01-18_10.53.36_pip_errors.log
Some possibly relevant errors from pip install:
ERROR: Ignored the following versions that require a different python version: 0.64.1 Requires-Python >=3.9,<3.12; 0.64.2 Requires-Python >=3.9,<=3.12; 0.64.3 Requires-Python >=3.9,<=3.12; 0.65.0 Requires-Python >=3.9,<=3.12
ERROR: Could not find a version that satisfies the requirement osxphotos==0.64.2 (from versions: 0.10.4, 0.10.5, 0.11.0, 0.12.1, 0.12.2, 0.12.3, 0.13.0, 0.14.0, 0.14.2, 0.14.4, 0.14.6, 0.14.8, 0.14.14, 0.14.21, 0.15.0, 0.17.2, 0.17.6, 0.18.0, 0.19.0, 0.20.0, 0.21.0, 0.21.1, 0.21.2, 0.21.5, 0.21.6, 0.22.0, 0.22.4, 0.22.5, 0.22.7, 0.22.9, 0.22.10, 0.22.12, 0.22.13, 0.22.16, 0.22.17, 0.22.21, 0.22.23, 0.23.1, 0.23.3, 0.23.4, 0.24.1, 0.24.2, 0.25.0, 0.25.1, 0.26.0, 0.26.1, 0.27.0, 0.27.1, 0.27.3, 0.27.4, 0.28.1, 0.28.2, 0.28.3, 0.28.4, 0.28.5, 0.28.6, 0.28.7, 0.28.8, 0.28.9, 0.28.10, 0.28.13, 0.28.14, 0.28.16, 0.28.17, 0.28.18, 0.28.19, 0.29.0, 0.29.1, 0.29.2, 0.29.5, 0.29.8, 0.29.9, 0.29.12, 0.29.13, 0.29.14, 0.29.16, 0.29.17, 0.29.18, 0.29.19, 0.29.22, 0.29.23, 0.29.24, 0.29.25, 0.29.26, 0.29.28, 0.29.29, 0.29.30, 0.30.0, 0.30.1, 0.30.2, 0.30.3, 0.30.4, 0.30.5, 0.30.6, 0.30.7, 0.30.9, 0.30.10, 0.30.12, 0.30.13, 0.31.0, 0.32.0, 0.33.0, 0.33.2, 0.33.3, 0.33.5, 0.33.7, 0.33.8, 0.34.0, 0.34.1, 0.34.2, 0.34.3, 0.34.4, 0.34.5, 0.35.0, 0.35.1, 0.35.2, 0.35.3, 0.35.4, 0.35.5, 0.35.6, 0.35.7, 0.36.0, 0.36.1, 0.36.2, 0.36.4, 0.36.5, 0.36.6, 0.36.7, 0.36.8, 0.36.9, 0.36.10, 0.36.11, 0.36.13, 0.36.14, 0.36.15, 0.36.16, 0.36.17, 0.36.18, 0.36.19, 0.36.20, 0.36.21, 0.36.22, 0.36.23, 0.36.24, 0.36.25, 0.37.0, 0.37.1, 0.37.2, 0.37.3, 0.37.4, 0.37.5, 0.37.6, 0.37.7, 0.38.0, 0.38.2, 0.38.3, 0.38.4, 0.38.5, 0.38.6, 0.38.7, 0.38.8, 0.38.9, 0.38.11, 0.38.12, 0.38.13, 0.38.14, 0.38.15, 0.38.16, 0.38.17, 0.38.18, 0.38.19, 0.38.20, 0.38.21, 0.38.22, 0.39.0, 0.39.1, 0.39.2, 0.39.3, 0.39.4, 0.39.5, 0.39.6, 0.39.7, 0.39.8, 0.39.9, 0.39.10, 0.39.11, 0.39.12, 0.39.13, 0.39.15, 0.39.16, 0.39.17, 0.39.18, 0.39.19, 0.39.20, 0.39.21, 0.39.22, 0.39.23, 0.39.24, 0.39.25, 0.40.0, 0.40.1, 0.40.2, 0.40.3, 0.40.5, 0.40.6, 0.40.7, 0.40.8, 0.40.9, 0.40.10, 0.40.11, 0.40.12, 0.40.13, 0.40.14, 0.40.16, 0.40.17, 0.40.18, 0.40.19, 0.41.0, 0.41.1, 0.41.2, 0.41.3, 0.41.4, 0.41.5, 0.41.6, 0.41.7, 0.41.8, 0.41.9, 0.41.10, 0.41.11, 0.42.0, 0.42.1, 0.42.2, 0.42.3, 0.42.4, 0.42.6, 0.42.8, 0.42.9, 0.42.11, 0.42.12, 0.42.13, 0.42.14, 0.42.15, 0.42.17, 0.42.19, 0.42.20, 0.42.22, 0.42.23, 0.42.24, 0.42.26, 0.42.27, 0.42.28, 0.42.30, 0.42.31, 0.42.34, 0.42.35, 0.42.36, 0.42.37, 0.42.38, 0.42.39, 0.42.40, 0.42.41, 0.42.42, 0.42.43, 0.42.44, 0.42.45, 0.42.46, 0.42.51, 0.42.52, 0.42.54, 0.42.57, 0.42.58, 0.42.59, 0.42.60, 0.42.61, 0.42.62, 0.42.63, 0.42.64, 0.42.65, 0.42.66, 0.42.67, 0.42.69, 0.42.70, 0.42.71, 0.42.72, 0.42.73, 0.42.74, 0.42.77, 0.42.78, 0.42.79, 0.42.80, 0.42.82, 0.42.83, 0.42.84, 0.42.85, 0.42.86, 0.42.87, 0.42.88, 0.42.89, 0.42.90, 0.42.91, 0.42.92, 0.42.93, 0.42.94, 0.43.0, 0.43.1, 0.43.3, 0.43.4, 0.43.5, 0.43.6, 0.43.7, 0.43.8, 0.43.9, 0.44.0, 0.44.1, 0.44.2, 0.44.3, 0.44.4, 0.44.5, 0.44.6, 0.44.7, 0.44.8, 0.44.9, 0.44.10, 0.44.11, 0.44.12, 0.44.13, 0.45.0, 0.45.2, 0.45.3, 0.45.4, 0.45.5, 0.45.6, 0.45.7, 0.45.8, 0.45.10, 0.45.11, 0.45.12, 0.46.0, 0.46.1, 0.46.5, 0.46.6, 0.47.0, 0.47.1, 0.47.2, 0.47.3, 0.47.4, 0.47.5, 0.47.6, 0.47.7, 0.47.8, 0.47.9, 0.47.10, 0.47.11, 0.47.12, 0.47.13, 0.48.0, 0.48.1, 0.48.2, 0.48.3, 0.48.4, 0.48.5, 0.48.6, 0.48.7, 0.48.8, 0.49.0, 0.49.1, 0.49.2, 0.49.3, 0.49.4, 0.49.5, 0.49.6, 0.49.7, 0.49.8, 0.49.9, 0.50.0, 0.50.1, 0.50.2, 0.50.3, 0.50.4, 0.50.5, 0.50.6, 0.50.7, 0.50.8, 0.50.9, 0.50.10, 0.50.11, 0.50.12, 0.50.13, 0.51.0, 0.51.1, 0.51.2, 0.51.3, 0.51.4, 0.51.5, 0.51.6, 0.51.7, 0.51.8, 0.52.0, 0.53.0, 0.54.0, 0.54.1, 0.54.2, 0.54.3, 0.54.4, 0.55.0, 0.55.1, 0.55.2, 0.55.3, 0.55.4, 0.55.5, 0.55.6, 0.55.7, 0.56.0, 0.56.1, 0.56.2, 0.56.3, 0.56.4, 0.56.5, 0.56.6, 0.56.7, 0.57.0, 0.57.1, 0.57.2, 0.57.3, 0.58.0, 0.58.1, 0.58.2, 0.59.0, 0.59.1, 0.59.2, 0.59.3, 0.60.0, 0.60.1, 0.60.2, 0.60.3, 0.60.4, 0.60.5, 0.60.6, 0.60.7, 0.60.8, 0.60.9, 0.60.10, 0.61.0, 0.62.0, 0.62.1, 0.62.2, 0.62.3, 0.63.0, 0.63.1, 0.63.2, 0.63.3, 0.63.4, 0.63.5, 0.64.0, 0.66.0, 0.67.0, 0.67.1, 0.67.2, 0.67.3)
ERROR: No matching distribution found for osxphotos==0.64.2
Error installing osxphotos from spec 'osxphotos==0.64.2'.❯ pipx uninstall osxphotos
uninstalled osxphotos! ✨ 🌟 ✨
❯ pipx install --python python3.11 osxphotos==0.64.2
installed package osxphotos 0.64.2, installed using Python 3.11.7
These apps are now globally available
- osxphotos
done! ✨ 🌟 ✨
❯ osxphotos -v
Traceback (most recent call last):
File "/Users/rajscode/.local/bin/osxphotos", line 5, in <module>
from osxphotos.__main__ import cli_main
File "/Users/rajscode/.local/pipx/venvs/osxphotos/lib/python3.11/site-packages/osxphotos/__init__.py", line 13, in <module>
from .exifwriter import ExifWriter
File "/Users/rajscode/.local/pipx/venvs/osxphotos/lib/python3.11/site-packages/osxphotos/exifwriter.py", line 19, in <module>
from .exportoptions import ExportOptions
File "/Users/rajscode/.local/pipx/venvs/osxphotos/lib/python3.11/site-packages/osxphotos/exportoptions.py", line 12, in <module>
from .phototemplate import RenderOptions
File "/Users/rajscode/.local/pipx/venvs/osxphotos/lib/python3.11/site-packages/osxphotos/phototemplate.py", line 15, in <module>
from textx import TextXSyntaxError, metamodel_from_file
File "/Users/rajscode/.local/pipx/venvs/osxphotos/lib/python3.11/site-packages/textx/__init__.py", line 2, in <module>
from textx.metamodel import metamodel_from_file, metamodel_from_str
File "/Users/rajscode/.local/pipx/venvs/osxphotos/lib/python3.11/site-packages/textx/metamodel.py", line 17, in <module>
from .registration import LanguageDesc, metamodel_for_language
File "/Users/rajscode/.local/pipx/venvs/osxphotos/lib/python3.11/site-packages/textx/registration.py", line 6, in <module>
import pkg_resources
ModuleNotFoundError: No module named 'pkg_resources'To confirm python3.12 and pipx 1.4.3 and latest working again: Installing osxphotos latest version ❯ pipx uninstall osxphotos
uninstalled osxphotos! ✨ 🌟 ✨
❯ pipx install osxphotos
installed package osxphotos 0.67.3, installed using Python 3.12.1
These apps are now globally available
- osxphotos
done! ✨ 🌟 ✨
❯ osxphotos -v
osxphotos, version 0.67.3
Python 3.12.1 (main, Dec 7 2023, 20:45:44) [Clang 15.0.0 (clang-1500.1.0.2.5)]
macOS 14.2.1, arm64To confirm older Python3.11 instance and pipx 1.4.3 working with latest version of osxphotos also. ❯ pipx uninstall osxphotos
uninstalled osxphotos! ✨ 🌟 ✨
❯ pipx install --python python3.11 osxphotos
installed package osxphotos 0.67.3, installed using Python 3.11.7
These apps are now globally available
- osxphotos
done! ✨ 🌟 ✨
❯ osxphotos -v
osxphotos, version 0.67.3
Python 3.11.7 (main, Dec 4 2023, 18:10:11) [Clang 15.0.0 (clang-1500.1.0.2.5)]
macOS 14.2.1, arm64 |
Beta Was this translation helpful? Give feedback.
-
|
I don't know how to debug this as it sounds like a pipx issue. It worked for me with python 3.11.6. You could use the prebuilt binary: https://github.com/RhetTbull/osxphotos/releases/tag/v0.64.2 |
Beta Was this translation helpful? Give feedback.
-
|
Thx. I grabbed v0.64.3 instead as it has an arm64 binary (0.64.2 only has the x86_64 version). Running the export with profile on. |
Beta Was this translation helpful? Give feedback.
-
|
v0.64.3 with profile on took ~ 2hr3mins. Full details further below. Comparison of runs: ~ 2hrs 44min on 0.67.3 (NO profile) v0.64.3 with profile details: /Users/rajscode/Downloads/osxphotos -v
osxphotos, version 0.64.3
Python 3.11.6 (v3.11.6:8b6ee5ba3b, Oct 2 2023, 11:18:21) [Clang 13.0.0 (clang-1300.0.29.30)]
macOS 14.2.1, arm64/Users/rajscode/Downloads/osxphotos --profile export /Volumes/WDSSD2TB/photo_library_export/photo_library_backup/ --ramdb --download-missing --use-photokit --load-config /Volumes/WDSSD2TB/photo_library_export/osxphotos_backup.toml > profile_v0_64_2_2024_01_18.txt
2024-01-18 13:54:10,587 - root - WARNING - photosdb.py - 151 - WARNING: This module has only been tested with macOS versions [10.12, 10.13, 10.14, 10.15, 10.16, 11.0, 11.1, 11.2, 11.3, 11.4, 11.5, 11.6, 11.7, 12.0, 12.1, 12.2, 12.3, 12.4, 12.5, 12.6, 13.0, 13.1, 13.2, 13.3, 13.4, 13.5, 14.0]: you have Darwin, OS version: 14.2
Profiling...
Profile sort_stats order: ['cumulative']
Exporting 210985 photos to /Volumes/WDSSD2TB/photo_library_export/photo_library_backup...
Exporting 210985 photos ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 100% 0:00:00
Processed: 210985 photos, exported: 0, updated: 2, skipped: 263424, updated EXIF data: 0, missing: 75, error: 0
Elapsed time: 2:02:26
Cleaning up /Volumes/WDSSD2TB/photo_library_export/photo_library_backup
Deleted: 27442 files, 0 directories
Profiling completed
10504430869 function calls (10200786536 primitive calls) in 7935.326 seconds |
Beta Was this translation helpful? Give feedback.
-
|
Thanks again for details. Let me know if you want me to try anything on my side.
Insert itself won't benefit from index(es) on the table(s). At times on large data imports one may drop the indexes totally and do the bulk inserts and then build the index post insert. Def loss in speed for the insert time due to index is made up for if doing lots of where statements for select / delete / update.
I definitely have seen when Backblaze is active scanning for new files or backing up it impacts speed due to contention on the drive(s)
Thank you!
Am exporting (update actually) ~ 211,000 photo/videos from my Apple Photos Library (1.66 TB). The photos library, the export directory and the export SQLite export db and SQLite report db are all stored on the same drive - external USB C connected SSD drive (not Thunderbolt 3 or 4). I could consider moving the .osxphotos and report_backup files to another drive (internal SSD for the Macbook Pro 14") but definitely have intentionally left the Apple Photos Library and the osxphotos exported files on the same externally connected SSD for file space usage benefits due to apfs. My directory structure is as below. In the yyyy-mm directory could have a few files or 5000 files: yyyy dir
|--> yyyy-mm dir
|--> individual media file
|--> xmp filemy .toml for the export. [export]
update = true
cleanup = true
report = "/Volumes/WDSSD2TB/photo_library_export/osxphotos_report_backup.sqlite"
append = true
db = "/Volumes/WDSSD2TB/Macbook_Pro_14_PhotosLibrary_Full_Originals_Backup/Photos Library.photoslibrary"
tmpdir = "/Volumes/WDSSD2TB/photo_library_export/osxphotos_tmp"
directory = "{created.year}/{created.year}-{created.mm}"
sidecar = [ "xmp",]
sidecar_drop_ext = true
person_keyword = true
favorite_rating = true
keyword_template = [
"Album: {folder_album}",
"{label}",
"{favorite?is_favorite}",
"is_{media_type}",
]
My .osxphotos_export.db is 2.2 GB currently. The osxphotos_report_backup.sqlite is 4 GB |
Beta Was this translation helpful? Give feedback.
-
True but one of the things I think is taking time is that the history feature maintains two tables: the first contains the path to the file being exported with an associated ID which is the foreign key into the actual history table which includes a record for every action ever taken by osxphotos on a file. This avoids repeating the path for every history record. But it must first check if the path exists in the path table and if not, add it. This means a lookup every time an insert is being done in the history table and the path table lacked an index. And the history is written for every file processed as part of the export set (even missing files). I'll do more testing but I think just adding the index here will help. The history table itself will grow indefinitely every time update is run so the data stored there is intentionally minimized. It doesn't need to be indexed as retrieving history is not something done during export, only when using |
Beta Was this translation helpful? Give feedback.
-
|
I ran the export again with the index enabled and it ran in 2h10min so close to the 2h5min baseline. I then disabled all the history code and ran again and this time it was 2h8min so it seems the bottleneck does not appear to be the history code. I'll push a release this weekend that adds the index to the history table then see how that performs for you. I think I don't yet have enough data to determine if it's worth adding a flag to disable history. |
Beta Was this translation helpful? Give feedback.
-
I checked the code and this is the correct behavior (at least "as designed" in the code). The version info specified by |
Beta Was this translation helpful? Give feedback.
-
|
|
Beta Was this translation helpful? Give feedback.
-
Wondering if something change in 0.67.4 vs prior release as it seems to have speed up quite significantly again. Didn't see anything in the release notes. Ran my standard weekly export job and it took only 1hr 26mins. I noticed the starting PROCESSING sequence prior to the EXPORTING portion seem to go much quicker vs 0.67.3 release (i.e. 1 min to get to the EXPORT vs ~ 5-7 mins to get to the EXPORT). UPDATE: Took a quick peek at the diff between 67.4 vs .3 and saw there are exportdb performance updates that were included. Also confirmed that my export database was upgraded. Export database version: 9.1, created by osxphotos version 0.67.4 ... (could it be useful to output to stdout anytime an upgrade is done to the export database. Otherwise one doesn't see that until the next time they run osxphotos export) Thx Latest osxphotos export Run Output Details
osxphotos, version 0.67.4 ---> Export Completed ... photos_library_export_to_disk_launchdjob scheduled job finished running. The job log at: /tmp/com.rajscodephotobackup.photos_library_export_to_disk_launchdjob.log.txt |
Beta Was this translation helpful? Give feedback.
-
Good catch. I thought I'd previously merged that fix in 0.67.3. I 'll update the release notes.
This does happen if you use Thanks for the PR on the comment, I've merged it. |
Beta Was this translation helpful? Give feedback.
-
I'm trying to use osxphotos to sync photos with a network storage. The initial sync worked very well, but updates take practically as much time as the initial sync. The library is stored locally on an external drive on a Mac and in iCloud (but iCloud synch does not work, so syncing via osxphotos, plus embedding metadata is a nice benefit!). Here is what I am doing:
osxphotos export '/Volumes/photos/icloud export' --directory '{created.year}/{created.month}' --force-update --cleanup --skip-original-if-edited --download-missing --touch-file --exiftool --exiftool-option -m --keyword-template '{label}' --exiftool-merge-keywords --exiftool-merge-persons --person-keyword --album-keyword --favorite-rating --exportdb '/Users/username/tmp/osxphotos_export.db'
It's a 110K file library but only about 100-300/week are added or updated. It took 10 hours for the initial export, which seems ok, but about 8 hours to update on a 10Gbit network with SSDs on both ends - seems slow. The goal is to keep a copy of the Photos library on a network drive, update files whenever file or metadata is changed, and delete files that are deleted in the master library.
Is there any way to speed up the updates?
I tried --only-new but that ends up deleting all previously exported files (oops) leaving literally only new ones :). My understanding is that --only-new without --cleanup will not remove any deleted files, and I'm not sure if it will update updated files. The documentation does not sound like it would.
I'm thinking if there is no full solution, I can do a workaround - run the slow update using the command line above weekly to monthly and running quick daily updates using the following:
osxphotos export '/Volumes/photos/icloud export' --directory '{created.year}/{created.month}' --force-update --only-new --skip-original-if-edited --download-missing --touch-file --exiftool --exiftool-option -m --keyword-template '{label}' --exiftool-merge-keywords --exiftool-merge-persons --person-keyword --album-keyword --favorite-rating --exportdb '/Users/username/tmp/osxphotos_export.db'
(w/o --clean-up but with --only-new).
Am I missing a better solution?
Beta Was this translation helpful? Give feedback.
All reactions