Skip to content

Conversation

@uzadude
Copy link
Collaborator

@uzadude uzadude commented Dec 21, 2022

Summary

playing around with other things I happened to reproduce #67

Detailed Description

How was it tested?

added unit test

@uzadude
Copy link
Collaborator Author

uzadude commented Dec 21, 2022

@eyala / @shay1bz - if you want to look at it.. we get this error org.apache.avro.AvroRuntimeException: java.io.IOException: Block read partially, the data may be corrupt when I run the test TestAvroPartialBlock with N=100000. with N=10000 it doesn't happen..

@shay1bz
Copy link
Collaborator

shay1bz commented Dec 21, 2022

I am limited to my mobile phone for the next few days :D Will try to debug it when I'm back 👍🏼

* avro btree - set sync interval to max

* remove redundat configuration

* fix invalid interval

* try increasing build memory heap

* relax sync interval

* set avro sync interval configurable

* take avro sync interval from hadoop configurations, plus some cleanup

---------

Co-authored-by: shay.elbaz <[email protected]>
@eyala
Copy link
Collaborator

eyala commented Mar 15, 2023

I'm going to take all of this PR other than the commit almost support non-partitioned tables and release a 0.5.x version with it - that's where we did some full scale testing on it. Afterwards we can determine whether to merge all of it to main

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants