Unsupported NumPy features and other differences w.r.t. NumPy

[ev-br]

• We only aim to support numeric dtypes, which are undestood by pytorch. This rules out

  • object arrays
  • datetimes
  • strings, chars and void dtypes
  • structured dtypes and recarrays
  • np.longdouble and np.clongdouble (a.k.a. np.float128 and np.complex256, respectively)

• ndrray subclasses are out of scope.

• masked arrays are out of scope

• numpy polynomials are out of scope, both np.poly1d, np.polynomial

• __array_function__ protocol is out of scope. This way, non-default like=... arguments raise.

• __array_interface__ is out of scope

• ndarray.ctypes attribute not supported

• Negative strides: tnp.flip and slicing with negative step return a copy.

---

These differences exist currently, but might be fixable if desired:

• We do not distinguish between 0D arrays and scalars. That is, tnp.float32(3) creates a zero-dim array.

  • One corollary is that scalars never silently decay into Python scalars, as NumPy scalars do in some situations:

In [1]: np.int32(2) * [1, 2, 3]         # scalar decays to a python int
Out[1]: [1, 2, 3, 1, 2, 3]

In [2]: np.asarray(2) * [1, 2, 3]     # zero-dim array is an array-like
Out[2]: array([2, 4, 6])

In our implementation, np.int32(2) behaves identically to np.asarray(2).

• We do not implement value-based casting. This will be deprecated in NumPy 2.0 as per NEP 50.
  

• __array_wrap__ protocol is currently not implemented.

• gufunc machinery is not implemented, e.g. axes=[(n,k),(k,m)->(n,m)] arguments of ufunc objects.

• ufunc methods (np.add.reduce etc) are not implemented.

• Fortran ordered arrays in general, and order="CFKA" in various creation functions are not implemented.

• numpy.linalg, handles zero-size arrays (sort of) uniformly, and pytorch doesn't handle these at all. We do not currently implement it.

• various estimators for the np.histogram bin selection are not implemented.

• nout=2 ufuncs out1=..., out2=... positional arguments do not work (out=tuple kwargs work)

• sorting/ordering of complex data: numpy defines some ordering of complex values, pytorch errors out; we follow pytorch; relevant functions are min/max, argmin/argmax, sort and searchsorted. cf min/max for complex inputs #67 for discussion.

• tril_indices_from/triu_indices_from return tensors rather than list of tuples to avoid a graph break

[rgommers]

__array_wrap__ protocol is out of scope. This way, non-default like=... arguments raise.

This is something that torch.Tensor itself uses - should that really be out of scope? It's unrelated to like= arguments AFAIK, that'd be __array_function__.

[ev-br]

Thanks, edited! Have to admit, I'm not up-to-date on various __array_dunder__ protocols. Not sure which of them does what, thus do not have a sensible suggestion on which ones we want here.

[rgommers]

Is there any test case like:

import torch
import numpy as np
from numpy.testing import assert_raises

t1 = torch.arange(3)
t2 = torch.ones(3)
t3 = np.add(t1, t2)
assert isinstance(t3, torch.Tensor)

# Test mixing torch and numpy tensors
x1 = np.full((3,), 2)
assert isinstance(t1 + x1, torch.Tensor)  # calls numpy, which is happy to accept tensors
assert_raises(TypeError, lambda f: x1 + t1)  # calls torch, which doesn't accept numpy arrays

That requires __array_wrap__. We don't want to encourage mixing tensors and arrays too much, but this one thing is used annoyingly often. So if it's possible to support with a minor amount of effort, then that's probably worth it.

[ev-br]

I'm not quite sure what is the desired behavior in your snippet above if import as np is replaced by the torch_np input, could you clarify? I'm probably just being dense.

On a tangentially related note, is there a way to control how numpy arrays and our wrapper arrays mix:

In [28]: tnp.array([1, 2, 3]) + np.array([4, 5, 6])              # tnp.array wins over
Out[28]: array_w([5, 7, 9])

In [29]: np.array([1, 2, 3]) + tnp.array([4, 5, 6])
Out[29]: array([array_w(5), array_w(7), array_w(9)], dtype=object)     #  ugh, object arrays! 

[rgommers]

I'm not quite sure what is the desired behavior in your snipped above if import as np is replaced by the torch_np input, could you clarify? I'm probably just being dense.

The exact same - you either get a torch.Tensor back (from numpy), or you get an exception (from pytorch).

On a tangentially related note, is there a way to control how numpy arrays and our wrapper arrays mix:

There is, e.g. through implementing __array__ to control how wrapper ndarrays get turned into numpy ndarrays (or raise). This mix should never happen though?

[ev-br]

Re your example. Is the expected behavior python/numpy/pytorch version dependent? (Sincerely hope it shouldn't be!). Here's what I get locally with x1 and t from #73 (comment):

In [38]: x1 + t1
---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
Input In [38], in <cell line: 1>()
----> 1 x1 + t1

TypeError: Concatenation operation is not implemented for NumPy arrays, use np.concatenate() instead. Please do not rely on this error; it may not be given on all Python implementations.

In [39]: t1 + x1
Out[39]: tensor([2, 3, 4])

In case it matters,

In [40]: np.__version__
Out[40]: '1.24.1'

In [41]: torch.__version__
Out[41]: '1.13.0'

In [42]: import sys; sys.version
Out[42]: '3.9.0 | packaged by conda-forge | (default, Nov 26 2020, 07:57:39) \n[GCC 9.3.0]'

[rgommers]

No, I just edited the wrong line during our call. Fixed now by swapping x1 and t1`.

[ev-br]

The current behavior is that the wrapper ndarray wins over in both __add__ and __radd__:

In [1]: import torch_np as tnp

In [2]: import torch

In [3]: torch.ones(3) + tnp.ones(3)
Out[3]: array_w([2., 2., 2.], dtype=float64)

In [4]: tnp.ones(3) + torch.ones(3)
Out[4]: array_w([2., 2., 2.], dtype=float64)

Here I think what happens is that torch.Tensor does not recognize tnp.ndarrays and gives up; tnp.ndarray, in turn, does tnp.asarray on the other argument, which happily accepts a Tensor. I'd think it's the behavior we want, no?