[Java] Exception when trying to load next batch while reading a parquet file

[vinayvenk]

Describe the bug, including details regarding any error messages, version, and platform.

Getting this exception when trying to load next batch while reading a parquet file.
The parsing works if the batch size is big enough to process all the parquet contents in one shot. But if I try to give a smaller batch size , the code breaks giving the below exception

java.lang.IllegalArgumentException: should have as many children as in the schema: found 0 expected 8
at org.apache.arrow.util.Preconditions.checkArgument(Preconditions.java:282)
at org.apache.arrow.vector.VectorLoader.loadBuffers(VectorLoader.java:127)
at org.apache.arrow.vector.VectorLoader.load(VectorLoader.java:84)
at org.apache.arrow.c.Data.importIntoVectorSchemaRoot(Data.java:334)
at org.apache.arrow.dataset.jni.NativeScanner$NativeReader.loadNextBatch(NativeScanner.java:151)

Component(s)

Java

[amoeba]

Hi @vinayvenk, what Arrow Java version are you on? Can you share the code you ran, ideally with code that can generate the data causing this?

[vinayvenk]

GM @amoeba it is pretty much standard code that i got from the example and I tried using 16.0.1, 16.0.0 and 15.0.1

String uri = parquetFile.toURI().toString();
ScanOptions options = new ScanOptions(/* batchSize */ 32768);
try (BufferAllocator allocator = new RootAllocator();
DatasetFactory datasetFactory = new FileSystemDatasetFactory(allocator, NativeMemoryPool.getDefault(),
FileFormat.PARQUET, uri);
Dataset dataset = datasetFactory.finish();
Scanner scanner = dataset.newScan(options);
ArrowReader reader = scanner.scanBatches()) {
int batchCount = 0;
while (reader.loadNextBatch()) {
try (VectorSchemaRoot root = reader.getVectorSchemaRoot()) {
//function to create csv data
createCSV()
}
it fails when it tries to load the second batch.

[majin1102]

Is there any progress on this issue?

[9aman]

@vinayvenk was a fix made for this ?

[krishan1390]

This happens specifically when reading a struct column. For primitive columns, loadNextBatch() works fine

[krishan1390]

This is mitigated by not closing the VectorSchemaRoot instance before calling loadNextBatch(). I guess closing it isn't required because the previous batches VectorSchemaRoot is updated in place.

Somehow this issue only occurs for struct columns though

[amoeba]

Thanks for the interest here.

I'm not familiar with that APIs but I notice that our documentation shows processing the batches with VectorUnloader. Can someone who can reproduce this issue check that this isn't the problem? From https://arrow.apache.org/java/current/dataset.html:

List<ArrowRecordBatch> batches = new ArrayList<>();
        while (reader.loadNextBatch()) {
            try (VectorSchemaRoot root = reader.getVectorSchemaRoot()) {
                final VectorUnloader unloader = new VectorUnloader(root);
                batches.add(unloader.getRecordBatch());
            }
        }

        // do something with read record batches, for example:
        analyzeArrowData(batches);

Otherwise, would someone who can reproduce this be willing to create a PR with a failing test?

[lidavidm]

You aren't supposed to close the root; if you want to save the batches you should use VectorUnloader indeed.