skip reprocessing of entire rows based on finegrained lineage information #1145
Conversation
There was a problem hiding this comment.
Thanks for having a try on this!
This is something really touching the core engine and multiple components, so its complexity is more than many other issues (that's why I didn't put a "help wanted" label on it).
I shared more thoughts on this thread:
#1057 (comment)
Please let me know if you have any thoughts. Thanks!
| let mut lineage_fp = Fingerprinter::default(); | ||
| lineage_fp = lineage_fp.with(&flow_inst.export_ops)?; | ||
| lineage_fp = lineage_fp.with(&flow_schema.schema)?; | ||
| let lineage_fingerprint = lineage_fp.into_fingerprint(); |
There was a problem hiding this comment.
This will miss certain information, like the spec changes (e.g. chunk size is changed for SplitRecursively), or behavior version change.
It will also be affected unnecessarily by some information, e.g. a field name (e.g. users may rename products to files in this example), and also ordering of ops (users may reorder some functions which will affect fields ordering in schema, they may also reordering collectors and export ops).
|
Thanks @georgeh0, I’ve gone through that thread. Implementing it fully would require a broader refactor, and I won’t be able to take that on at the moment. Please feel free to unassign me so that someone else can pick it up if needed. |
|
got it, thanks a lot @Davda-James , i'll close this PR for now. Thank you so much for your contribution and discussion! |
3 participants
Description
skip reprocessing of entire rows based on finegrained lineage information
Closes Issue: #1057