Feat: Adding Linear Algebra Dot operation support

[SwayamInSync]

This PR contributes as follows:

• Ship the dot method within package that supports following operations
  • vector-vector dot product
  • matrix-vector dot product
  • matrix-matrix multiplication
• Optimized Linear Algebra ops supported by the QBLAS on x86-64 Linux and ARM machine. On windows it fallbacks to naive implementation due to QBLAS incompatibility with MSVC
• Test Suite to validate dot products between inputs

Images below are the performance comparison

Machine: x86-64 Linux with 96 cores

Machine: MacOS-Silicon (ARM) with 8 cores

To compile without QBLAS set DISABLE_QUADBLAS as CFLAGS and CXXFLAGS

[SwayamInSync]

Ahh forgot to update the general CI, or should we remove the quaddtype from there, given that build_wheels.yml has those same checks in its CI with more strict and on all platforms

[juntyr]

Would the new functionality only be accessed through the dot function or is there a way to call that automatically when using numpy dot, matmul, etc?

[ngoldbaum]

Unfortunately no, not easily. We'd need to add a new dtype hook to the DType API in NumPy. Worth doing though! See numpy/numpy#28516 which adds a hook for sorting.

[ngoldbaum]

(unless I'm missing an existing hook - @seberg might know better)

[seberg]

I think the most interesting thing is actually @ which goes via matmul which is a ufunc (like vecdot, vecmat, matvec). And adding ufuncs is straight forward of course!

The old style ArrFuncs also has a PyArray_DotFunc, but it is relatively useless in practice, because it implements a vector dot. arr.dot(arr2) will use it but just end up being slow, I suspect.

(One fun thing is, as a "work-around" you could try to implement that PyArray_DotFunc and possibly raise an error to use the ufunc instead. -- although the function would be called many times.)

Adding new slots similar to the sort way makes sense. Although, I think for arr.dot(arr2) it may be more relevant to see if doesn't make sense to change it to call into matmul?
(Which may need some care, because I am not sure what the current paths are for objects, i.e. if those call back into OBJECT_dot, but I don't think so.)

[SwayamInSync]

I see, so if anyways we create a slot, the implementation need to be in done for the quaddtype.
Now with @seberg 's comment I remember trying NPY_DT_PyArray_ArrFuncs_dotfunc but it also wasn't supported.

The current dot implementation (shipped with package) handles the vector-vector, gemv or gemm as per the input dimensions.

It would be good to have a slot parallel to np.dot (like it also picks the operation based on input dimensions)

[SwayamInSync]

Ahh forgot to update the general CI, or should we remove the quaddtype from there, given that build_wheels.yml has those same checks in its CI with more strict and on all platforms

@ngoldbaum thoughts on this?

[seberg]

It would be good to have a slot parallel to np.dot (like it also picks the operation based on input dimensions)

Actually, I think we should probably not do this, unless arr.dot() really can't call into one of the existing ufuncs. I.e. yes, I am all for new slots, but if ufuncs work, that is better (even a new ufunc is likely better)!
(It does behave slightly differently, but I suspect one can map it.)

And the first step, is to implement those ufuncs, I think.