What is the best way to integrate with Frigate with low latency? Looking to rewrite DoubleTake / CompreFace

[bertybuttface]

Hello,

What is currently the best way to integrate with Frigate in a low latency way? Integrating directly with ZeroMQ?

DoubleTake is sadly no longer maintained and CompreFace (which for me gives the best performance) is a bit of a mess and also unmaintained. I'd be looking to implement a new take on this, the features I'd be after are:

1. Entity Recognition (starting with faces but recognising the same dog, same car, same cat would be interesting too).
2. Action Recognition (is the recognised entity cooking, eating, watching TV, sleeping).
3. Location Detection (where is the recognised entity in relation to the rest of the room e.g. at the sink, by the door)

I accept that some of Action Recognition is being done by the ML captions now but I'd be specifically looking to tie this in with the Entity Recognition e.g. "Alice is reading the newspaper" in the arm chair or "Bob is chopping vegetables at the counter".

This feels achievable with current tech but I'd like to do this once, properly without having to switch methods part way through.

If ZeroMQ is the correct way to do this is the ZeroMQ interface documented? Is it stable? Is it worth bothering with over MQTT?

I remember there being some discussion on this previously but I can't find it despite searching.

Any tips / insight greatly appreciated.

Thanks a lot

[hawkeye217]

At the moment, the HTTP API and MQTT are the most supported and best documented interfaces to Frigate's data.

In coming versions, a post-processing "plugin" system will be available as the preferred method for tasks like you're describing.

It should also be noted that Frigate 0.16 will support face recognition natively.