Copyless mir.ndslice and numpy.ndarray integration

[9il]

No description provided.

[ariovistus]

Which direction are you envisioning? Seems to me going numpy.ndarray -> ndslice makes sense as long as the numpy array is C contiguous, it's just extract the appropriate pointers to the data and put them in the d structure. I don't know off the top of my head how feasible going the other way would be.

memory management - Going from python to D should be fine, pyd keeps an extra reference so python won't reclaim it. Actually, it seems like pyd keeps references on both sides, so this shouldn't be an issue. Until we start supporting non GC allocated memory.

[9il]

Which direction are you envisioning?

Both.

[9il]

ndslice part of work is done in mir.ndslice.connect.cpython

[ariovistus]

Buffer interface seems like the right way to go, pyd doesn't have a way to wrap buffer interface implementations, so that needs to be added. You might consider making Py_buffer a template parameter in mir, since you're going to be getting a pyd Py_buffer struct, and the contents of the struct may differ depending on the version of python (looks like it hasn't changed since 2.5, but you never know what will happen in the future).

Buffer interface requires functions getbuffer and releasebuffer to be implemented. I would expect to be able to wrap d methods/functions of signatures

int my_getbuffer(Py_buffer* buffer);
int my_getbuffer(Py_buffer* buffer, int flags);
int my_getbuffer(T myObj, Py_buffer* buffer, int flags);
void my_releasebuffer();
void my_releasebuffer(Py_buffer* buffer);
void my_releasebuffer(T myObj, Py_buffer* buffer);

but we'll see.

2.7 and earlier will also require some additional functions, hopefully I can just wrap getbuffer.

I would expect this to be part of pyd's wrap_class functionality, which would probably require making wrapper classes on the mir side. Usage would probably look something like

wrap_class!(MyClass,
        Buffer!(MyClass.getbuffer, MyClass.releasebuffer),
    )();

That would handle the use case of mir -> python

now how about python -> mir
using PyObject_GetBuffer with fromPythonBuffer should work, I'll want to wrap the annoying parts of that.

use case:

InterpContext context;
auto result = context.my_numpy_array.to!YourMirType;
do_something(result);

provided the appropriate converter is defined. I notice fromPythonBuffer is copying pointers, which raises concerns about python destroying the buffer while you're using it. Hm. Could just require the user to keep a reference to a PydObject somewhere. That might work sometimes. What does mir use for memory allocation? Pyd is pretty coupled to garbage collector, and being able to support other schemes is probably going to require some significant redesign.

use case:

InterpContext context;
context.do_something = do_something;
context.py_stmts("do_something(my_numpy_array");

if do_something's signature has a YourMirType parameter, same as above, conversion is automatic, otherwise you'll have to add it manually somewhere. Reference counting is still the problem.

[9il]

What does mir use for memory allocation?

Mir does not specify allocations. By default users use GC allocated memory (slice function). Probably we do not need a special functions for ndslice, but a few clever functions that adds, removes roots from the GC.

[ariovistus]

ok. I think we'll have to constrain this to GC allocated memory only, since that's all pyd supports right now

[mw66]

I want to ask a related question: how can I pass a D (native) array to the Python side? I don't care copyless or not, as long as I can pass it.

Any example code to show how this can be done?

Thanks.

#139