Missing Photos / Video - Disparate count results depending on parameters used with osxphotos

[rajscode]

Still in awe of osxphotos functionality combined with APFS in enabling so much flexibility for many of us. Thanks again for this awesome tool and making the code available for us to dig into.

Been curious as to why # of "missing" photos/videos are different when run against the same (full resoluton) PhotosDB that is in showing in sync with iCloud. I suspect it probably has to do with the nuances that cover:

• Old style Shared Albums. Pictures in those by default don't download into your own library.
  One has to manually select the picture(s) and have them download.
• Newer shared library behavior. I haven't enabled this yet to my knowledge though I am on Ventura (13.31.1).
• Shared with You. Photos / Videos shared by others via iMessages / Messages App. Whether one has selected to "save" them from the Messages app manually or has turned on auto save all incoming photo video items. (prior to running commands below, I'd gone to the Shared With You in Photos. Filtered for Unsaved Items. Selected All. Saved.)
• Burst pictures
• Live pictures

Issued 4 different osxphotos commands and got 4 different results. When I do the math around the results -- I can see some of it adding up via taking into account the different skipped amounts between RUN A and B. But Run C and D are different (click on each below to see details of the command and its results).

RUN A - osxphotos export --download-missing --use-photokit ==> 13 missing

osxphotos export photo_library_backup/ --download-missing --use-photokit --load-config osxphotos_backup.toml

Exporting 207671 photos to /Volumes/WDSSD2TB/photo_library_export/photo_library_backup...
Exporting 207671 photos ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 100% 0:00:00
Processed: 207671 photos, exported: 1, updated: 2, skipped: 257806, updated EXIF data: 0, missing: 13, error: 0
Elapsed time: 5:38:40
Cleaning up /Volumes/WDSSD2TB/photo_library_export/photo_library_backup
Deleted: 0 files, 0 directories

RUN B - osxphotos export ==> 7463 missing

osxphotos export photo_library_backup/ --load-config osxphotos_backup.toml

Exporting 207671 photos to /Volumes/WDSSD2TB/photo_library_export/photo_library_backup...
Exporting 207671 photos ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 100% 0:00:00
Processed: 207671 photos, exported: 0, updated: 1, skipped: 250357, updated EXIF data: 0, missing: 7463, error: 0
Elapsed time: 1:29:40
Cleaning up /Volumes/WDSSD2TB/photo_library_export/photo_library_backup
Deleted: 1 file, 0 directories

RUN C - osxphotos query --missing ==> 70 missing

osxphotos query --missing > listing_shared_2023_05_15.csv

Results in 70 missing photos info being returned with the following attributes:

Attribute	True / Yes	False / Blank
iscloudasset	TRUE: 69	FALSE: 1
ismissing	TRUE: 67	FALSE: 3
shared	TRUE: 66	FALSE: 4
incloud	TRUE: 3	BLANK: 67

NOTE: 3 items ismissing = FALSE also are the ONLY ones to be Album = BLANK and incloud = TRUE

Overall I'd expected this to be 67 missing but 70 were returned via the --missing command.

Extract from RUN C of Results returned:

RUN D - osxphotos repl ==> 67 missing

osxphotos repl --db "/Volumes/WDSSD2TB/Macbook_Pro_14_PhotosLibrary_Full_Originals_Backup/Photos Library.photoslibrary"

Getting photos
Found 196702 photos in 1.71 seconds

missing = [p for p in photos if p.ismissing]
...
...

Attribute	Count
len(missing)	67
len(uuid)	196702
len(cloudasset)	196701
len(incloud)	155864
len(notincloud)	40838
len(shared)	40837
len(burst)	1065
len(live_photo)	12163

Items I am going to attempt to investigate further by looking at the codebase to understand why there is a difference in the "missing" based on the parameter used. The commands were one after another upon completion of prior one.

• RUN A: osxphotos export --download-missing --use-photokit --load-config

  reporting back missing as 13. Interestingly the skipped is: 257806
  Processed: 207671 photos, exported: 1, updated: 2, skipped: 257806, updated EXIF data: 0, missing: 13, error: 0
  Elapsed time: 5:38:40

• RUN B: osxphotos export --load-config
  reporting back missing as 7463. The skipped here is 250357.
  Processed: 207671 photos, exported: 0, updated: 1, skipped: 250357, updated EXIF data: 0, missing: 7463, error: 0
  Elapsed time: 1:29:40

  When I do the following calculation:
  (RUN A Skipped - RUN B Skipped) + RUN A Missing + RUN A Exported (or RUN B Updated) you get == RUN B Missing total.

  Curious about the disparity in the missing reported between RUN A and RUN B. The totals of all objects added up seems to be proper though. Just how they are bucketed as MISSING or SKIPPED seems different based on parameters used during export.

• RUN C: osxphotos query --missing

  Reports back 70 items (or 67 if we exclude the aforementioned 3 ismissing = FALSE)
  Why are 3 items being returned that have the ismissing field set as FALSE

• RUN D: osxphotos repl --db "/Volumes/WDSSD2TB/Macbook_Pro_14_PhotosLibrary_Full_Originals_Backup/Photos Library.photoslibrary"

  reporting back 67 items seems consistent with RUN C if RUNC returned 67 instead of 70.

[RhetTbull]

Thanks for the detailed notes. I am traveling and can't take a look just yet but will try to explore when I have time. This has been something that needs fixing for some time. osxphotos "sees" for example the shared photos which do not show up in the Photos total. It does not yet support the "shared with me" photos (but these do show up in the total count), etc. I welcome any feedback on how to make these numbers more clear to the user.