Simple heuristics to identify frames of interaction

[sfmig]

Is your feature request related to a problem? Please describe.
People often use pose estimation output to analyse social interaction. We could implement functionality in movement that helps the user identify frames in which the animals are likely interacting.

Describe the solution you'd like
Some simple heuristics that may be useful (feel free to expand here or in the comments).

• Find frames in which the animal centroids are closer than a certain threshold.

Describe alternatives you've considered
\

Additional context
\

[roaldarbol]

Could also incorporate how they are oriented relative to each other. Since you have the posterior2anterior vector, it might be relevant to point out when they are facing each other (in which case the most frontal coordinate may be more interesting than the centroid), or parallel to each other (common in Drosophila mating displays if I remember correctly).

[talmo]

or parallel to each other (common in Drosophila mating displays if I remember correctly)

The males actually do a ton of circling around the female if she's immobile, so that vector would span all the angles (in addition to chasing which will look more parallel).

---

We really like using shapely for this kind of stuff. Having reimplemented way too many computational geometry routines during my PhD (specifically to quantify fly courtship), I really like deferring to shapely whenever possible.

For example, for social investigation across mice, what we do is compute the convex hull of the head keypoints and the body keypoints (separately), then use shapely.shortest_line to find the nearest point of interaction between the geometries. This lets you not have to rely on having a keypoint on the specific part of the body that they might be investigating with (or being investigated).

Here's a colab with an example pipeline.

[roaldarbol]

I like that idea! Also because shapely is built on top of GEOS - and GEOS is supported in R via both the geos package and the more comprehensive sf (simple features) package. That way, we could probably find some community-based examples (e.g. like @talmo's pipelines), and implement them as generalised functions across both movement and animovement.

EDIT: I wonder whether it would mostly make sense to implement some of the more objective things, such as e.g. a function that, given two geometries (could be body-related keypoints and points that make up the wall) and compute the nearest distance between the two geometries. And then leave it to the user to have their own ( achine learning) methods for assigning descriptions such as "chase", "close to wall", etc.

[niksirbi]

We like shapely too, and it absolutely makes sense to rely on it for all sorts of geometric calculations.
We were thinking of adding it as a dependency soon, primarily to support workflows with ROIs (arena, objects in the arena, etc.), but Talmo just made me realise it's also very useful for animal interaction stuff.

[roaldarbol]

I feel like possibly organising a hackathon/workshop could be cool for coming up with and implementing a bunch of geometrical/relational functions 😅

[niksirbi]

Would be cool, but given the geographical distribution of interested parties, would be hard to organise in person, and It's tricky to capture the hackathon spirit in online events. Perhaps attached to some conference where many interested people are likely to be is the most feasible?

[talmo]

Cosyne?

[roaldarbol]

Cosyne?

Would be mega awesome! I'll have finished my PhD then (probably viva around those dates), and don't have a position or funding lined up, so I can't go unfortunately. :-(

[niksirbi]

CoSyne would be great, but I haven't yet decided whether I'm attending or not.