Error when writing to fluss: IndexOutOfBoundsException

[gongzexin]

Search before asking

• I searched in the issues and found nothing similar.

Fluss version

0.6.0 (latest release)

Please describe the bug 🐞

While writing a large amount of data(for example, after writing 10 million+ records to the flux table, consume this table and write to another table) into fluss, the client receives the following error

2025-03-26 01:51:46,314 ERROR com.alibaba.fluss.server.replica.ReplicaManager [] - Error put records to local kv on replica TableBucket{tableId=43, bucket=0}
java.lang.IndexOutOfBoundsException: capacity: 262144, index: 262144, put length: 1
at com.alibaba.fluss.memory.MemorySegment.put(MemorySegment.java:324) ~[fluss-server-0.6.0.jar:0.6.0]
at com.alibaba.fluss.record.ChangeTypeVectorWriter.writeChangeType(ChangeTypeVectorWriter.java:43) ~[fluss-server-0.6.0.jar:0.6.0]
at com.alibaba.fluss.record.MemoryLogRecordsArrowBuilder.append(MemoryLogRecordsArrowBuilder.java:169) ~[fluss-server-0.6.0.jar:0.6.0]
at com.alibaba.fluss.server.kv.wal.ArrowWalBuilder.append(ArrowWalBuilder.java:47) ~[fluss-server-0.6.0.jar:0.6.0]
at com.alibaba.fluss.server.kv.KvTablet.lambda$putAsLeader$0(KvTablet.java:338) ~[fluss-server-0.6.0.jar:0.6.0]
at com.alibaba.fluss.utils.concurrent.LockUtils.inLock(LockUtils.java:41) ~[fluss-server-0.6.0.jar:0.6.0]
at com.alibaba.fluss.utils.concurrent.LockUtils.inWriteLock(LockUtils.java:64) ~[fluss-server-0.6.0.jar:0.6.0]
at com.alibaba.fluss.server.kv.KvTablet.putAsLeader(KvTablet.java:271) ~[fluss-server-0.6.0.jar:0.6.0]
at com.alibaba.fluss.server.replica.Replica.lambda$putRecordsToLeader$10(Replica.java:847) ~[fluss-server-0.6.0.jar:0.6.0]
at com.alibaba.fluss.utils.concurrent.LockUtils.inLock(LockUtils.java:41) ~[fluss-server-0.6.0.jar:0.6.0]
at com.alibaba.fluss.utils.concurrent.LockUtils.inReadLock(LockUtils.java:54) ~[fluss-server-0.6.0.jar:0.6.0]
at com.alibaba.fluss.server.replica.Replica.putRecordsToLeader(Replica.java:833) ~[fluss-server-0.6.0.jar:0.6.0]
at com.alibaba.fluss.server.replica.ReplicaManager.putToLocalKv(ReplicaManager.java:894) [fluss-server-0.6.0.jar:0.6.0]
at com.alibaba.fluss.server.replica.ReplicaManager.putRecordsToKv(ReplicaManager.java:430) [fluss-server-0.6.0.jar:0.6.0]
at com.alibaba.fluss.server.tablet.TabletService.putKv(TabletService.java:143) [fluss-server-0.6.0.jar:0.6.0]
at jdk.internal.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) ~[?:?]
at jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source) ~[?:?]
at java.lang.reflect.Method.invoke(Unknown Source) ~[?:?]
at com.alibaba.fluss.rpc.netty.server.RequestProcessor.processRequest(RequestProcessor.java:108) [fluss-server-0.6.0.jar:0.6.0]
at com.alibaba.fluss.rpc.netty.server.RequestProcessor.run(RequestProcessor.java:84) [fluss-server-0.6.0.jar:0.6.0]
at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) [?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) [?:?]
at java.lang.Thread.run(Unknown Source) [?:?]

Solution

No response

Are you willing to submit a PR?

• I'm willing to submit a PR!

[gongzexin]

Adding the following parameters to the tablet-server can help alleviate this problem.
server.buffer.memory-size: 1gb
server.buffer.page-size: 1mb

[swuferhong]

Adding the following parameters to the tablet-server can help alleviate this problem. server.buffer.memory-size: 1gb server.buffer.page-size: 1mb

Hi, @gongzexin. did you using a Flink job to write into fluss? If so, you can try increasing the off-heap memory for the Flink sink. The Fluss writer requires a larger off-heap memory to handle large batches of data.

[gongzexin]

Adding the following parameters to the tablet-server can help alleviate this problem. server.buffer.memory-size: 1gb server.buffer.page-size: 1mb

Hi, @gongzexin. did you using a Flink job to write into fluss? If so, you can try increasing the off-heap memory for the Flink sink. The Fluss writer requires a larger off-heap memory to handle large batches of data.

@swuferhong Hi, thanks your reply. In this flink job, the taskmanager has 3.2gb of managed memory. Do you think I need to increase it further?

[swuferhong]

Adding the following parameters to the tablet-server can help alleviate this problem. server.buffer.memory-size: 1gb server.buffer.page-size: 1mb

Hi, @gongzexin. did you using a Flink job to write into fluss? If so, you can try increasing the off-heap memory for the Flink sink. The Fluss writer requires a larger off-heap memory to handle large batches of data.

@swuferhong Hi, thanks your reply. In this flink job, the taskmanager has 3.2gb of managed memory. Do you think I need to increase it further?

How about flink parallelism and fluss table buckets? If you have a bucket number like 512, I recommend to set a large flink parallelism like 128/256 instead of giving much memory.

[wuchong]

@swuferhong this is a bug in server side. The IndexOutOfBoundsException is thrown in KvTablet. The root cause is that currently, we write ChangeType into the first memory segment page and doesn't support dynamically writing to second page if the first page is full. That's why the IndexOutOfBoundsException is thrown when number of changelogs is larger than the bit size of each page.