Underflow bugfix & fix off-by-one error in the texture height calculation #1909
+28
−4
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
hannes-vernooij
force-pushed
the
lut1d-off-by-one
branch
3 times, most recently
from
24a3863 to
b078247
Compare
hannes-vernooij
force-pushed
the
lut1d-off-by-one
branch
6 times, most recently
from
0b2fee9 to
d03be01
Compare
hannes-vernooij
changed the title
hannes-vernooij
changed the title
hannes-vernooij
force-pushed
the
lut1d-off-by-one
branch
from
d03be01 to
f4b201b
Compare
It should be max index / max size because 4096 / 4096 + 1 = 2, which means that two rows are allocated when a LUT1D with a size of 4096 fits in a 1D texture of 4096 pixels. Signed-off-by: Hannes Vernooij <[email protected]>
(131071U / 4096U) + 1 = 32 Signed-off-by: Hannes Vernooij <[email protected]>
hannes-vernooij
force-pushed
the
lut1d-off-by-one
branch
from
f4b201b to
9005a12
Compare
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
It should bemax index / max sizebecause4096 / 4096 + 1 = 2, which means that two rows are allocated when a LUT1D with a size of 4096 fits in a 1D texture of 4096 pixels.EDIT:
In the initial commit I made the mistake of not taking into account that the last value of a row is repeated in the first value of the next row, this is now fixed. I validated the result by running a shaper LUT with a size of 4096, 4097, 8191 and 8192. I then applied the transform LUT on
syntheticChart.01.exrand validated a 0 pixel difference.Underflow bugfix
This also seems to solve an existing underflow bug that appeared with a 1D LUT with a size of 8191. When the height is calculated the value is not ceiled, which results in
8191 / 4096 = 2, while it should be3.This caused an underflow later in the code in:
unsigned long missingEntries = width * height - (unsigned long)paddedChannel.size();where
width * heightis less than(unsigned long)paddedChannel.size();causing an underflow that results in a very high value that is iterated.