-
Notifications
You must be signed in to change notification settings - Fork 20
Auto rechunk to enable blockwise reduction #380
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
base: main
Are you sure you want to change the base?
Conversation
bcae992
to
0c0b0e5
Compare
f0af7ee
to
20e6ed8
Compare
Done when 1. `method` is None 2. Grouping and reducing by a 1D array We gate this on fractional change in number of chunks and change in size of largest chunk. Closes #359
20e6ed8
to
d9a7ab5
Compare
new_input_chunks = chunks | ||
|
||
# FIXME: this should be unnecessary? | ||
labels = factorize_((labels,), axes=())[0] |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
TODO: get rid of this line
@@ -119,6 +119,15 @@ | |||
# _simple_combine. | |||
DUMMY_AXIS = -2 | |||
|
|||
# Thresholds below which we will automatically rechunk to blockwise if it makes sense | |||
# 1. Fractional change in number of chunks after rechunking | |||
BLOCKWISE_RECHUNK_NUM_CHUNKS_THRESHOLD = 0.25 |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
TODO: these should probably be in some kind of "options" context manager
if ascending: | ||
return bool((arr[:-1] <= arr[1:]).all()) | ||
else: | ||
return bool((arr[:-1] >= arr[1:]).all()) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
should add a test for descending
Done when
method
is Noneby
is sorted.by
tooWe gate this on fractional change in number of chunks and change in size of largest chunk.
Closes #359