Sending large data streams _into_ Python

[tonybaloney]

When sending lots of data from Python into C# there are options:

• Using bytes or bytearray
• Streaming data using a generator
• Using a Buffer protocol object such as bytes or ndarray and then casting into a Span

All are performant options for when Python returns a lot of binary data.

However, we don't really have many options for sending a lot of data from C# back into Python efficiently.

The fastest option right now is using a writable buffer but that isn't obvious. Creating Python lists from value types like int or string from C# > Python is slow.

• Supporting a lower-level type like array
• Creating a custom Python type that can chunk/stream something iterable in .NET

Related user requests:

• 2D Array Transfering from C# to Python #347
• Is support for mapping typing.BinaryIO to Stream or ReadOnlyMemory<byte> under consideration? #371

[marmic7]

I ran experiments through the Benchmarking suite to test sending large data streams from Python to C#. For byte and bytearray types, I compared the base performance with converting these types to Buffer types (from collections.abc), and then converting them back to the original types in C# using AsByteReadOnlySpan() or AsByteSpan().

Notes:

• Sending data as bytearrays appears to be more performant over bytes.
• Sending datatypes as buffers improves performance for the bytes datatype but not bytearray
• When using buffers for bytes datatypes, the performance with bytearrays is similar