Speed-up multiple exports

[nicad]

Hi,
I'm experimenting with doing all the following exports daily:

• the entire library
• only favorites
• only deleted
• only new
• in the future I'd also like to export per tagged person albums

So it looks like te following bash script:

APPLE_PHOTO_EXPORT_ALL="$LOCAL_MEDIA/Photos/Apple/All"
function osxphotos-export-all {
    mkpath "$APPLE_PHOTO_EXPORT_ALL"
    osxphotos-export $APPLE_PHOTO_EXPORT_ALL \
        --no-progress --export-by-date  --exiftool --update --touch-file \
        --cleanup
}

APPLE_PHOTO_EXPORT_NEW="$LOCAL_MEDIA/Photos/Apple/New"
function osxphotos-export-new {
    mkpath "$APPLE_PHOTO_EXPORT_NEW"
    osxphotos-export $APPLE_PHOTO_EXPORT_NEW \
        --no-progress --export-by-date  --exiftool --update --touch-file \
        --only-new
    # --cleanup   problem is that if it's ran twice then all photos are deleted
}

# Keep the current set of deleted photos, once purged they will stop showing up here
APPLE_PHOTO_EXPORT_DELETED="$LOCAL_MEDIA/Photos/Apple/Deleted"
function osxphotos-export-deleted {
    mkpath "$APPLE_PHOTO_EXPORT_DELETED"
    osxphotos-export $APPLE_PHOTO_EXPORT_DELETED \
        --no-progress --export-by-date  --exiftool --update --touch-file \
        --deleted-only --cleanup
}

# if a photo is removed from favorites the local photo is modified
APPLE_PHOTO_EXPORT_FAVORITES="$LOCAL_MEDIA/Photos/Apple/Favorites"
function osxphotos-export-favorites {
    mkpath "$APPLE_PHOTO_EXPORT_FAVORITES"
    osxphotos-export $APPLE_PHOTO_EXPORT_FAVORITES \
        --no-progress --export-by-date  --exiftool --update --touch-file \
        --favorite --cleanup
}

My question: this results into maintaining an osxphotos export database for each export and it takes a while. Is it possible to do this with a single osxphotos database or any faster ?

[RhetTbull]

Yes, this is possible using a custom template function. In effect, you'd use your custom template function for the --directory template and let it decide where to put each photo. The deleted and favorite are easy because the template function receives a PhotoInfo object for the photo being exported that can be examined to determine if photo is in trash or is a favorite. The "New" export is more challenging because the template function won't know if the photo is a new photo. This is solved in the example below by using a key-value store (a simple database) to track which photos have previously been exported.

What is your intent for cleaning up the "New" folder? You currently don't run --cleanup on it which means New will just grow infinitely and contain a copy of all photos in a single directory. Is this what you want? Do you clean it up with a separate script?

Assuming you want to keep the files in New/ from being deleted by cleanup, then with the following template function, you should also create a file .osxphotos_keep in the $LOCAL_MEDIA/Photos/Apple/ directory. The contents should be:

/New

This instructs --cleanup to not delete any files in the New/ directory.

One note: this workflow requires the export database reside in "$LOCAL_MEDIA/Photos/Apple/" as one export database will manage all these directories. Unfortunately there's no easy way to combine the export databases you have or migrate them to the new directory so you'll have to effectively do a 1-time "new export" to re-export everything to build the database. I recommend starting with a clean directory structure (maybe archive the current folder). If you are well versed in SQL then I could walk you through what needs to be be done to try to patch the database but that is likely more trouble than just doing a fresh export.

The following command assumes you've saved the following template function as "nicad.py" in the directory from whence you run the export command. Obviously you can change this.

Here's the command:

osxphotos export "$LOCAL_MEDIA/Photos/Apple/" --deleted --directory "{function:nicad.py::nicad}" --update --cleanup --verbose

This uses the function nicad() in nicad.py as a custom template for the --directory value. Here's the contents of nicad.py file:

""" Example showing how to use a custom function for osxphotos {function} template 
    Use:  osxphotos export /path/to/export --filename "{function:/path/to/nicad.py::nicad}"

    You may place more than one template function in a single file as each is called by name using the {function:file.py::function_name} format

    This template function implements a solution to the question asked in:
    https://github.com/RhetTbull/osxphotos/discussions/1328
"""
from __future__ import annotations

import datetime
import os
import pathlib
import sys
from typing import Optional, Union

from osxphotos import PhotoInfo
from osxphotos.cli.kvstore import kvstore
from osxphotos.export_db import ExportDB
from osxphotos.phototemplate import RenderOptions

# store for UUIDs of previously exported photos
KVSTORE = "osxphotos_exported_photos"

# pre-load the kvstore so it's ready when the function is called with all previously exported photos
# this could be moved to separate script that is run once to pre-populate the kvstore
# or you could delete this part of the code after the first run
# this is only needed if exporting into a previously exported directory
# otherwise the kvstore will be updated automatically during export
LOCAL_MEDIA = os.getenv("LOCAL_MEDIA")
if not LOCAL_MEDIA:
    print("LOCAL_MEDIA environment variable not set!")
    sys.exit(1)

EXPORT_DIR = pathlib.Path(LOCAL_MEDIA) / "Photos" / "Apple"
if not os.path.isdir(EXPORT_DIR):
    print("Export directory does not exist!")
    sys.exit(1)

EXPORT_DB = EXPORT_DIR / ".osxphotos_export.db"
if EXPORT_DB.exists():
    # get list of all previously exported photo uuids
    uuids = ExportDB(EXPORT_DB, EXPORT_DIR).get_previous_uuids()
else:
    uuids = []

# write list of uuids to a key/value store so we can check if a photo has been previously exported
kv = kvstore(KVSTORE)
for uuid in uuids:
    if uuid not in kv:
        kv[uuid] = datetime.datetime.now().isoformat()
kv.close()


def nicad(
    photo: PhotoInfo, options: RenderOptions, args: Optional[str] = None, **kwargs
) -> Union[list[str], str]:
    """example function for {function} template; intended to be used as a template function with --directory

    Args:
        photo: osxphotos.PhotoInfo object
        options: osxphotos.phototemplate.RenderOptions object
        args: optional str of arguments passed to template function
        **kwargs: not currently used, placeholder to keep functions compatible with possible changes to {function}

    Returns:
        str or list of str of values that should be substituted for the {function} template
    """

    directories = []

    if not photo.intrash:
        # every photo not in trash also gets exported to main "All" directory
        rendered, _ = photo.render_template("{created.year}/{created.mm}/{created.dd}")
        export_by_date = rendered[0]
        directories.append(f"All/{export_by_date}")

    if photo.intrash:
        directories.append("Deleted")

    if photo.favorite:
        directories.append("Favorites")

    # no way of knowing if photo is new or not so need to maintain state of new photos
    # this is done with a key/value store (simple SQLite database stored in ~/.local/share)
    # using a helper function provided by osxphotos
    kv = kvstore(KVSTORE)
    if photo.uuid not in kv:
        # store the date photo was exported so we can check if needed
        kv[photo.uuid] = datetime.datetime.now().isoformat()
        directories.append("New")

    return directories

[nicad]

Wow, thank you for the super quick and thorough answer. !
I'll need a little bit of time to digest it and try it out. It's not a problem to redo an export.

Yes the "New" concept is shacky, with --cleanup there is a chance to lose data if it's ran twice in a row before another script can do something. So I'll probably export the last X months of photos and re-import recent data maybe using the kvstore you suggested. I need to bake this some more.

[RhetTbull]

In place of "New" you could use --added-in-last 7d for example to keep a running export of last 7 days. You could arbitrarily use any almost any day/week/month value here, for example, --added-in-last "1 month". Coupled with --cleanup, this would keep one folder of just the most recent photos.

However, if intent is to do one single export, you wouldn't want to use --added-in-last as that limits the overall export. Instead, in the template function, you'd look at the date and if the delta is <= X days, add it to the "New" folder. Coupled with --cleanup, this would ensure the folder doesn't grow infinitely large and keeps only "X days" of images.