how to pass Mat or byte[] to python as np.typing.NDArray[np.uint8]

[deerchao]

Hi, I'm trying to use mediapipe from .net to estimate human body poses from images. Here's the python code:

import numpy as np
import mediapipe as mp
pose = mp.solutions.pose.Pose()

def mediapipe_body_pose(frame: np.typing.NDArray[np.uint8]) -> list[float]:
    keypoints: list[float] = []
    results = pose.process(frame)
    if results.pose_landmarks:
        for _, landmark in enumerate(results.pose_landmarks.landmark):
            keypoints.append(landmark.x)
            keypoints.append(landmark.y)
    return keypoints

Generated method signaure is like:

  IReadOnlyList<double> MediapipeBodyPose(PyObject frame)

Now I have an OpenCvSharp.Mat returned from VideoCapture.Read(), how do I pass it to generated method MediapipeBodyPose()?

I tried PyObject.From(mat), and get this:

System.InvalidCastException: 
Cannot convert Mat [ 960*606*CV_8UC3, IsContinuous=True, IsSubmatrix=False, Ptr=0x24daa6b1b90, Data=0x28e4788e080 ] to PyObject

[deerchao]

I figured it out.

def mediapipe_body_pose(width: int, height: int, channel: int, buffer: Buffer) -> list[float]:

    frame = np.ndarray((height, width, channel), np.uint8, buffer)

var buffer = new byte[mat.Width * mat.Height * mat.Channels()];
Marshal.Copy(mat.Data, buffer, 0, buffer.Length);
var landmarks = Program.Python
    .Infer()
    .MediapipeBodyPose(mat.Width, mat.Height, mat.Channels(), PyObject.From(buffer));

It would be nice to have a overload PyObject.From(Span<byte> data) to avoid the buffer allocating and copying.

[tonybaloney]

Did you see the details on the buffer protocol?

https://tonybaloney.github.io/CSnakes/buffers/

This is mostly for returning ndarrays and reading them as spans

[deerchao]

Yes, I saw it.

But didn't find out how to pass buffer allocated in C# to Python, untill I look through source code of PyObject.From(). The docs is about the other way around.

[tonybaloney]

The buffer API only gives access to allocated ndarrays created in Python. You cannot create them from C#. I recommend using a 2-D list of floats as the input signature then creating the array in the function. Have a look at the sample project for an example with skl