Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Remove byte order logic from half-transparency lookups #6872

Merged
merged 1 commit into from
Dec 19, 2023

Conversation

StephenCWills
Copy link
Member

I think I see two issues. First, here is where we draw the half-transparent rectangle.

const uint32_t v = *pix;
#if SDL_BYTEORDER == SDL_LIL_ENDIAN
*pix = lookupTable[v & 0xFFFF] | (lookupTable[(v >> 16) & 0xFFFF] << 16);
#else
*pix = lookupTable[(v >> 16) & 0xFFFF] | (lookupTable[v & 0xFFFF] << 16);
#endif

When we make use of this lookup table on big-endian systems, we swap the 16-bit values we pass into the lookup, but the 16-bit values themselves are in the system's native endian order. However, since we are interpreting the original pixel as a native-endian 32-bit value and also assigning *pix as a native-endian 32-bit value, I think it would be easiest to use native-endian order throughout. The little-endian logic doesn't swap any bytes so we can just remove the special case for big-endian systems.

And that just leaves us with making sure that we build the lookup table using native-endian order...

#if SDL_BYTEORDER == SDL_LIL_ENDIAN
const std::uint16_t index = i | (j << 8);
#else
const std::uint16_t index = j | (i << 8);
#endif
paletteTransparencyLookupBlack16[index] = paletteTransparencyLookup[0][i] | (paletteTransparencyLookup[0][j] << 8);

On big-endian systems, i is placed before j when computing index, but paletteTransparencyLookup[0][j] is placed before paletteTransparencyLookup[0][i] regardless of the endian order. This swaps the bytes on big-endian systems when you use this lookup table. Since we want to use native-endian order, we don't need to swap any bytes on any systems. Therefore, once again, we should be able to remove the special case for big-endian systems since we currently don't swap the bytes on little-endian systems.

This resolves #6871 (I assume)

@AJenbo AJenbo merged commit 4f8dc6c into diasurgical:master Dec 19, 2023
23 checks passed
@StephenCWills StephenCWills deleted the native-order-transparency branch December 19, 2023 23:14
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

[Issue Report]: possible big endian problem
3 participants