Skip to content

Some dtypes not supported by numba #521

@aulemahal

Description

@aulemahal

Describe the bug
Big-endian floats (and maybe others) are not supported by numba, which leads to TypingError on some sparse operations.

To Reproduce

import numpy as np
import sparse as sp

little = np.array([1, 1, 1, 1, 1, 1], dtype='<f8')
big = np.array([1, 1, 1, 1, 1, 1], dtype='>f8')

Anp = np.random.randint(100, size=(4, 6)) / 100
A = sp.COO.from_numpy(Anp)

A.dot(little) # works
A.dot(big) # fails

The other way around is the same;

Abig = sp.COO.from_numpy(Anp.astype('>f8'))

Abig.dot(little) # fails
Abig.dot(big) # fails

Expected behavior
I expected the operation to work in all cases.

Could on-the-fly byte swap be implemented? May be with a warning?

In this specific case, on-the-fly byte swapping is what numpy does:

r = Anp.dot(big)  
r.dtype # <f8, little-endian

For the case with Abig, where the sparse array itself is in a unsupported dtype, if not on-the-fly conversion is done, maybe there could be an issue raised, or at least a warning, on creation? To signify that the input array doesn't support the full features of sparse.

System

  • OS and version: Ubuntu 18.04
  • sparse version : 0.13.0
  • NumPy version : 1.21.2
  • Numba version : 0.53.1

Additional context

Activity

added
bugIndicates an unexpected problem or unintended behavior
on Oct 13, 2021
added and removed
bugIndicates an unexpected problem or unintended behavior
on Oct 14, 2021
hameerabbasi

hameerabbasi commented on Oct 14, 2021

@hameerabbasi
Collaborator

Hmm, this is an upstream Numba bug, as you already said. It's hard to detect the full scope of these kinds of things without try-catch.

hameerabbasi

hameerabbasi commented on Oct 14, 2021

@hameerabbasi
Collaborator

On-the-fly swapping could be done, in theory. That's something I would consider cleaner.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

      Development

      No branches or pull requests

        Participants

        @hameerabbasi@aulemahal

        Issue actions

          Some dtypes not supported by numba · Issue #521 · pydata/sparse