Draft bugfix: panic null value in dataset #91
Conversation
Chao-Ma5566
changed the title
There was a problem hiding this comment.
Proposal for additional data processing step
I propose adding an additional data processing step before constructing the tree in the SIGO workflow. Currently, the workflow follows this pattern:
- Input -> Tree Construction -> Tree
- Tree -> Aggregation -> Output
I suggest the following modification:
- Input -> Data Validation -> Validated Table
- Validated Table -> Tree Construction -> Tree
- Tree -> Aggregation -> Output
This adjustment offers several advantages:
-
Isolation of Error Handling: By separating the data validation step, we ensure that the rest of the processing is not cluttered with error handling logic. This promotes cleaner and more focused code for each stage of the workflow.
-
Early Pre-processing: Introducing a pre-processing step allows us to address data integrity issues upfront, improving the overall quality of the data fed into subsequent stages of the workflow.
I believe this change will enhance the maintainability and robustness of the SIGO system. Looking forward to your feedback and discussion on this proposal.
pkg/sigo/kdtree.go
Outdated
| less := func(i, j int) bool { | ||
| valueI, err := n.cluster[i].QuasiIdentifer() | ||
| if err != nil { | ||
| // Stocker l'erreur dans la variable globale |
There was a problem hiding this comment.
Yes, adding a step Data Validation is a better way to handling error. I will revert to the first commit just keep the venom test.
I propose adding an new interface dataValidator in case we use other type than float64 in the future. After by default we use float64DataValidator after source created to valide input data then we can focus in other step of the workflow.
Chao-Ma5566
changed the title
No description provided.