Stack overflow during gfx_FloodFill

[Bobb56]

Hello,
I am currently facing an issue with the gfx_FloodFill function.
I have a program that uses gfx_FloodFill, and sometimes while gfx_FloodFill is running, the calculator's RAM is cleared. I checked that the x and y coordinates are within the screen boundaries, and I also checked that the RAM Cleared happens in the gfx_FloodFill's code. So this is really a problem with gfx_FloodFill and not my program.

The CEMU console indicates that this is caused by a stack overflow (writing at address D19881).

[adriweb]

(The code in question is here, original report here)

[ZERICO2005]

Do you know what the screen looks like when this crash happens? gfx_FloodFill uses more memory if the area has a complex/fractal shape

[ZERICO2005]

It seems that gfx_FloodFill always allocates ~3224 bytes of stack. The stack size is usually around 4096 bytes, so if your program is already using more than ~800 bytes of stack then calling gfx_FloodFill could cause a stackoverflow to occur. You may have to roll your own flood-fill implementation (possibly using malloc to allocate memory)

[Bobb56]

Do you know what the screen looks like when this crash happens? gfx_FloodFill uses more memory if the area has a complex/fractal shape

The shape is not that complex, but as you say, it is likely that the amount of bytes allocated on the stack when I call gfx_FloodFill is relatively significant.

[ZERICO2005]

Some approaches discussed so far:

• Reducing stack usage in gfx_FloodFill
• Providing gfx_FloodFill_SetBuffer

It might be possible to have gfx_FloodFill use a variable amount of stack, instead of always allocating 3224 bytes. If we reduce the stack usage gfx_FloodFill from 3224 bytes to lets say 1000 bytes, then stack overflow issues would be less likely to occur. There are floodfill algorithms that use O(1) (constant) memory, but they may or may not perform as well. Another way to reduce stack usage would be to prove that gfx_FloodFill will never use more than N bytes of stack.

Since graphx functions cannot call or link to malloc, a gfx_FloodFill_SetBuffer(unsigned char* buf, size_t size) function can be provided instead. This would allow a user to supply their own arbitrary sized buffer (minimum of ~2048 bytes in size) to use as the stack for gfx_FloodFill. One possible implementation might be:

// passing in NULL uses the stack instead of a buffer
SetBuffer(NULL, 0)            // uses stack with the default allocation amount
SetBuffer(NULL, invalid_size) // ??
SetBuffer(NULL, valid_size)   // uses stack
SetBuffer(buf, invalid_size)  // falls back to using the stack
SetBuffer(buf, valid_size)    // uses the provided buffer

[mateoconlechuga]

There should be no "invalid size", the size should be checked in gfx_FloodFill