AI to detect similar images across collections

[glenrobson]

From the EuropeanaTech IIIF workshop

How do we imagine this? Incorporated into the IIIF Viewer, or usable in other contexts such as websites?
Maybe it’s finetuning of existing models? Maybe it can be used for metadata generation?
→ Maybe better to preprocess and incorporate the data into the source metadata
On the client side it might mean too heavy processing? How do we make it accessible for bigger and smaller institutions?
There is also the consideration of bias in labelling of the images.
Tools specifically for researchers
Making metadata collections based on image collections.
Processing the images and adding the results of the image recognition as an annotation layer.
Does tooling around AI for image recognition belong in IIIF? This is something to think about.
Maybe something around image recognition that clusters similar images and displays that data in the IIIF viewer.
Maybe the consideration is mostly for if this would make AI image recognition more accessible for institutions that don’t have the knowledge and capacity to do something with AI software.
AI can generate useful metadata to put items more into context, but it might not be always possible to incorporate this metadata into the source metadata (limitations of data models). Then it could make sense to have the generation process incorporated into IIIF as an annotation layer.
Europeana is working on improving low quality images with AI. Maybe that would be something to consider for including in IIIF tooling.

Possible tools:

• Wrapper around something like iiif-prezi3 to make harvesting images for AI easier.
• Something similar to https://github.com/orgs/IIIF-Commons/discussions/8 to help with harvesting
• A library with lists of manifests or API access to manifests from various collections.

[stephenwf]

We could have a set of embedding models, maybe starting with a small one that could be run inexpensively and even client side. Generating labels sounds like it could be more problematic than just a vector that could be used for image similarity.

Could we offer a cookbook entry on how to attach pre-generated embeddings, along with the model that was used to create them? It could also show how to differentiate between an embedding of the image itself (for image similarity comparison) and embeddings of the transcribed text.

Wrapper around something like iiif-prezi3 to make harvesting images for AI easier

On the flip-side, what patterns could we establish to ensure institutions are able to choose and/or rate limit these types of harvesting? Should any open source tools ensure they comply with IIIFs embedded licensing requirements?

[aisaac]

As I'm coming back to these notes, use case here is quite unclear to me. Which user scenario would we like to support?