fix: set finish before notifying load in LanceArrowWriter setFinished method

[ColdL]

Recently I was testing the integration between Lance and Spark.

I found that the Lance writer has a certain probability of hanging. After some troubleshooting, I discovered this is related to the LanceArrowWriter.setFinished method.

The original code appears to have a bug where it sets the finished status after notifying loadNextBatch, which could cause loadNextBatch to hang.

Root Cause

The ideal flow should be:

1. (thread 1) loadToken.release
2. (thread 1) finished = true
3. (thread 2) loadNextBatch
4. (thread 2) finished is true and count is 0 so return false

However, there's a chance it becomes:

1. (thread 1) loadToken.release
2. (thread 2) loadNextBatch
3. (thread 2) finished is false so return true and waiting
4. (thread 1) finished = false

If the second scenario occurs, thread 2 will hang indefinitely and cannot receive new notifications. jstack will show stacks hanging in LanceDataWriter.commit.

Reproduction

This issue is hard to reproduce. I encountered it in a very low-resource environment (Spark executor with only 1 core 4g) when creating a new table and writing 600 rows of data at once, where one column is a 1024-dimensional vector column.

It also occurs intermittently.

Further Confirmation

Although the current fix seems reasonable, I hope to get confirmation from maintainers to avoid introducing new unknown issues.

Any comments about this. @jackye1995

[jackye1995]

Nice, I think that explains https://github.com/lancedb/lance-spark/blob/main/lance-spark-base_2.12/src/test/java/com/lancedb/lance/spark/write/LanceArrowWriterTest.java#L38. Could you re-enable that test?

[ColdL]

DONE

[jackye1995]

looks like there are still some code style issues.

[ColdL]

DONE

[ColdL]

And, this commit might also be worth another look, it hasn't been merged yet 😉

@jackye1995