[Kernel][Clustering #4] add ClusteringUtils #4322
Open
+420
−0
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
KaiqiJinWow
force-pushed
the
stack/add_cluster_utils
branch
2 times, most recently
from
6b652bb to
7e9e1b6
Compare
KaiqiJinWow
commented
Mar 26, 2025
| @@ -18,6 +18,7 @@ | |||
| import static java.lang.String.format; | |||
|
|
|||
| import io.delta.kernel.exceptions.*; | |||
There was a problem hiding this comment.
To Reviewers,
This is a stack PR and please only review the latests commit which includes all the relevant code changes.
huan233usc
requested review from
vkorukanti,
huan233usc,
scottsand-db,
allisonport-db and
raveeram-db
KaiqiJinWow
force-pushed
the
stack/add_cluster_utils
branch
2 times, most recently
from
72dc84b to
37dbd23
Compare
vkorukanti
requested changes
Mar 27, 2025
Comment on lines
+82
to
+85
| if (!minValues.containsKey(column) | ||
| || !maxValues.containsKey(column) | ||
| || !nullCounts.containsKey(column)) { | ||
| throw DeltaErrors.missingColumnStatsForClustering(column, dataFileStatus); |
There was a problem hiding this comment.
is there a case where if all values are null, we don't have min and max?
There was a problem hiding this comment.
@raveeram-db do we have any utility that can validate stats are valid? @KaiqiJinWow is asking.
There was a problem hiding this comment.
Don't think we have any, I think we thought we'll leave it up to the engines to validate but we could perhaps refactor out basic checks like the one here into a helper?
KaiqiJinWow
force-pushed
the
stack/add_cluster_utils
branch
from
37dbd23 to
08de273
Compare
raveeram-db
reviewed
Mar 31, 2025
| throw DeltaErrors.missingFileStatsForClustering(clusteringColumns, dataFileStatus); | ||
| } | ||
|
|
||
| DataFileStatistics dataFileStatistics = dataFileStatus.getStatistics().get(); |
There was a problem hiding this comment.
Should we throw if missing? dataFileStatus.getStatistics().orElseThrow(..)
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Which Delta project/connector is this regarding?
Description
Split the main PR #4265 for faster review
This PR adds the ClusteringUtils which includes a few utils would could be used for clustering feature
How was this patch tested?
Unit tests, more tests would be added later in integration tests.
Does this PR introduce any user-facing changes?