Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

ArrowCube Format #5

Open
echeipesh opened this issue Oct 17, 2019 · 0 comments
Open

ArrowCube Format #5

echeipesh opened this issue Oct 17, 2019 · 0 comments
Labels
Epic

Comments

@echeipesh
Copy link
Collaborator

echeipesh commented Oct 17, 2019 •
edited
Loading

This epic covers developing arrow schema that is optimal for representing multi-dimensional rasters as tensors.

Design goals

  1. Records in this schema can accessed through zero-copy ndarray
  • This may require numpy extension
  1. Schema supports efficient dimension slicing by virtue of being paged and columnar
  • Arrow pages can map unto bands or other "dimension-slices"
  • It should be cheap to slice/select in at least one dimension of the array
  1. Records in this schema map well on common image encoding formats
  • pixel-interleave, band-interleave
  1. Some subset of GeoTrellis Tile interface or ND4J interface can be backed by these records
  • we're willing to give up efficient random access
  • we kind of get this one for "free", its not a big restriction

Its not clear that we can achieve all this so this is a wish-list that can be negotiated down
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
Epic
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@echeipesh