Skip to content

HIVE-29534: getColStatistics to use HMS NDV values for DATE/TIMESTAMP column types #6406

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Draft
konstantinb wants to merge 2 commits into
base: master
Choose a base branch
from
Draft

HIVE-29534: getColStatistics to use HMS NDV values for DATE/TIMESTAMP column types #6406

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
1f3b9fc
HIVE-29534: getColStatistics to use HMS NDV values forDATE/TIMESTAMP …
konstantinb Apr 2, 2026
801eaa5
HIVE-29534: regenerated impacted .out files
konstantinb Apr 2, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
48 changes: 24 additions & 24 deletions iceberg/iceberg-handler/src/test/results/positive/llap/iceberg_bucket_map_join_1.q.out
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -111,9 +111,9 @@ Stage-0
Stage-1
Map 1 llap
File Output Operator [FS_10]
Select Operator [SEL_9] (rows=30 width=520)
Select Operator [SEL_9] (rows=3 width=520)
Output:["_col0","_col1","_col2","_col3","_col4","_col5"]
Map Join Operator [MAPJOIN_45] (rows=30 width=336)
Map Join Operator [MAPJOIN_45] (rows=3 width=336)
BucketMapJoin:true,Conds:SEL_2._col0, _col1=RS_7._col0, _col1(Inner),Output:["_col0","_col1","_col2","_col3"]
<-Map 2 [CUSTOM_EDGE] llap
MULTICAST [RS_7]
Expand Down Expand Up @@ -175,19 +175,19 @@ Stage-0
Stage-1
Map 1 llap
File Output Operator [FS_14]
Select Operator [SEL_13] (rows=10 width=520)
Select Operator [SEL_13] (rows=3 width=520)
Output:["_col0","_col1","_col2","_col3","_col4","_col5"]
Map Join Operator [MAPJOIN_49] (rows=10 width=336)
Map Join Operator [MAPJOIN_49] (rows=3 width=336)
BucketMapJoin:true,Conds:SEL_2._col0, _col1=RS_11._col0, _col1(Inner),Output:["_col0","_col1","_col2","_col3"]
<-Reducer 3 [CUSTOM_EDGE] llap
MULTICAST [RS_11]
PartitionCols:_col1
Group By Operator [GBY_8] (rows=1 width=168)
Group By Operator [GBY_8] (rows=3 width=168)
Output:["_col0","_col1"],keys:KEY._col0, KEY._col1
<-Map 2 [SIMPLE_EDGE] llap
SHUFFLE [RS_7]
PartitionCols:_col0, _col1
Group By Operator [GBY_6] (rows=1 width=168)
Group By Operator [GBY_6] (rows=3 width=168)
Output:["_col0","_col1"],keys:date_col, decimal_col
Select Operator [SEL_5] (rows=3 width=168)
Output:["date_col","decimal_col"]
Expand Down Expand Up @@ -245,9 +245,9 @@ Stage-0
Stage-1
Map 1 llap
File Output Operator [FS_10]
Select Operator [SEL_9] (rows=30 width=520)
Select Operator [SEL_9] (rows=3 width=520)
Output:["_col0","_col1","_col2","_col3","_col4","_col5"]
Map Join Operator [MAPJOIN_45] (rows=30 width=336)
Map Join Operator [MAPJOIN_45] (rows=3 width=336)
Conds:SEL_2._col0, _col1=RS_7._col0, _col1(Inner),Output:["_col0","_col1","_col2","_col3"]
<-Map 2 [BROADCAST_EDGE] llap
BROADCAST [RS_7]
Expand Down Expand Up @@ -309,19 +309,19 @@ Stage-0
Stage-1
Map 1 llap
File Output Operator [FS_14]
Select Operator [SEL_13] (rows=10 width=520)
Select Operator [SEL_13] (rows=3 width=520)
Output:["_col0","_col1","_col2","_col3","_col4","_col5"]
Map Join Operator [MAPJOIN_49] (rows=10 width=336)
Map Join Operator [MAPJOIN_49] (rows=3 width=336)
Conds:SEL_2._col0, _col1=RS_11._col0, _col1(Inner),Output:["_col0","_col1","_col2","_col3"]
<-Reducer 3 [BROADCAST_EDGE] llap
BROADCAST [RS_11]
PartitionCols:_col0, _col1
Group By Operator [GBY_8] (rows=1 width=168)
Group By Operator [GBY_8] (rows=3 width=168)
Output:["_col0","_col1"],keys:KEY._col0, KEY._col1
<-Map 2 [SIMPLE_EDGE] llap
SHUFFLE [RS_7]
PartitionCols:_col0, _col1
Group By Operator [GBY_6] (rows=1 width=168)
Group By Operator [GBY_6] (rows=3 width=168)
Output:["_col0","_col1"],keys:date_col, decimal_col
Select Operator [SEL_5] (rows=3 width=168)
Output:["date_col","decimal_col"]
Expand Down Expand Up @@ -379,9 +379,9 @@ Stage-0
Stage-1
Map 1 vectorized, llap
File Output Operator [FS_54]
Select Operator [SEL_53] (rows=30 width=520)
Select Operator [SEL_53] (rows=3 width=520)
Output:["_col0","_col1","_col2","_col3","_col4","_col5"]
Map Join Operator [MAPJOIN_52] (rows=30 width=336)
Map Join Operator [MAPJOIN_52] (rows=3 width=336)
BucketMapJoin:true,Conds:SEL_51._col0, _col1=RS_49._col0, _col1(Inner),Output:["_col0","_col1","_col2","_col3"]
<-Map 2 [CUSTOM_EDGE] vectorized, llap
MULTICAST [RS_49]
Expand Down Expand Up @@ -443,19 +443,19 @@ Stage-0
Stage-1
Map 1 vectorized, llap
File Output Operator [FS_61]
Select Operator [SEL_60] (rows=10 width=520)
Select Operator [SEL_60] (rows=3 width=520)
Output:["_col0","_col1","_col2","_col3","_col4","_col5"]
Map Join Operator [MAPJOIN_59] (rows=10 width=336)
Map Join Operator [MAPJOIN_59] (rows=3 width=336)
BucketMapJoin:true,Conds:SEL_58._col0, _col1=RS_56._col0, _col1(Inner),Output:["_col0","_col1","_col2","_col3"]
<-Reducer 3 [CUSTOM_EDGE] vectorized, llap
MULTICAST [RS_56]
PartitionCols:_col1
Group By Operator [GBY_55] (rows=1 width=168)
Group By Operator [GBY_55] (rows=3 width=168)
Output:["_col0","_col1"],keys:KEY._col0, KEY._col1
<-Map 2 [SIMPLE_EDGE] vectorized, llap
SHUFFLE [RS_54]
PartitionCols:_col0, _col1
Group By Operator [GBY_53] (rows=1 width=168)
Group By Operator [GBY_53] (rows=3 width=168)
Output:["_col0","_col1"],keys:date_col, decimal_col
Select Operator [SEL_52] (rows=3 width=168)
Output:["date_col","decimal_col"]
Expand Down Expand Up @@ -513,9 +513,9 @@ Stage-0
Stage-1
Map 1 vectorized, llap
File Output Operator [FS_54]
Select Operator [SEL_53] (rows=30 width=520)
Select Operator [SEL_53] (rows=3 width=520)
Output:["_col0","_col1","_col2","_col3","_col4","_col5"]
Map Join Operator [MAPJOIN_52] (rows=30 width=336)
Map Join Operator [MAPJOIN_52] (rows=3 width=336)
Conds:SEL_51._col0, _col1=RS_49._col0, _col1(Inner),Output:["_col0","_col1","_col2","_col3"]
<-Map 2 [BROADCAST_EDGE] vectorized, llap
BROADCAST [RS_49]
Expand Down Expand Up @@ -577,19 +577,19 @@ Stage-0
Stage-1
Map 1 vectorized, llap
File Output Operator [FS_61]
Select Operator [SEL_60] (rows=10 width=520)
Select Operator [SEL_60] (rows=3 width=520)
Output:["_col0","_col1","_col2","_col3","_col4","_col5"]
Map Join Operator [MAPJOIN_59] (rows=10 width=336)
Map Join Operator [MAPJOIN_59] (rows=3 width=336)
Conds:SEL_58._col0, _col1=RS_56._col0, _col1(Inner),Output:["_col0","_col1","_col2","_col3"]
<-Reducer 3 [BROADCAST_EDGE] vectorized, llap
BROADCAST [RS_56]
PartitionCols:_col0, _col1
Group By Operator [GBY_55] (rows=1 width=168)
Group By Operator [GBY_55] (rows=3 width=168)
Output:["_col0","_col1"],keys:KEY._col0, KEY._col1
<-Map 2 [SIMPLE_EDGE] vectorized, llap
SHUFFLE [RS_54]
PartitionCols:_col0, _col1
Group By Operator [GBY_53] (rows=1 width=168)
Group By Operator [GBY_53] (rows=3 width=168)
Output:["_col0","_col1"],keys:date_col, decimal_col
Select Operator [SEL_52] (rows=3 width=168)
Output:["date_col","decimal_col"]
Expand Down
3 changes: 3 additions & 0 deletions ql/src/java/org/apache/hadoop/hive/ql/stats/StatsUtils.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -839,12 +839,14 @@ public static ColStatistics getColStatistics(ColumnStatisticsObj cso, String col
cs.setNumNulls(csd.getBinaryStats().getNumNulls());
} else if (colTypeLowerCase.equals(serdeConstants.TIMESTAMP_TYPE_NAME)) {
cs.setAvgColLen(JavaDataModel.get().lengthOfTimestamp());
cs.setCountDistint(csd.getTimestampStats().getNumDVs());
cs.setNumNulls(csd.getTimestampStats().getNumNulls());
Long lowVal = (csd.getTimestampStats().getLowValue() != null) ? csd.getTimestampStats().getLowValue()
.getSecondsSinceEpoch() : null;
Long highVal = (csd.getTimestampStats().getHighValue() != null) ? csd.getTimestampStats().getHighValue()
.getSecondsSinceEpoch() : null;
cs.setRange(lowVal, highVal);
cs.setBitVectors(csd.getTimestampStats().getBitVectors());
cs.setHistogram(csd.getTimestampStats().getHistogram());
} else if (colTypeLowerCase.equals(serdeConstants.TIMESTAMPLOCALTZ_TYPE_NAME)) {
cs.setAvgColLen(JavaDataModel.get().lengthOfTimestamp());
Expand All @@ -869,6 +871,7 @@ public static ColStatistics getColStatistics(ColumnStatisticsObj cso, String col
cs.setHistogram(csd.getDecimalStats().getHistogram());
} else if (colTypeLowerCase.equals(serdeConstants.DATE_TYPE_NAME)) {
cs.setAvgColLen(JavaDataModel.get().lengthOfDate());
cs.setCountDistint(csd.getDateStats().getNumDVs());
cs.setNumNulls(csd.getDateStats().getNumNulls());
Long lowVal = (csd.getDateStats().getLowValue() != null) ? csd.getDateStats().getLowValue()
.getDaysSinceEpoch() : null;
Expand Down
66 changes: 66 additions & 0 deletions ql/src/test/org/apache/hadoop/hive/ql/stats/TestStatsUtils.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -35,8 +35,12 @@
import org.apache.hadoop.hive.metastore.api.BooleanColumnStatsData;
import org.apache.hadoop.hive.metastore.api.ColumnStatisticsData;
import org.apache.hadoop.hive.metastore.api.ColumnStatisticsObj;
import org.apache.hadoop.hive.metastore.api.Date;
import org.apache.hadoop.hive.metastore.api.DateColumnStatsData;
import org.apache.hadoop.hive.metastore.api.DoubleColumnStatsData;
import org.apache.hadoop.hive.metastore.api.LongColumnStatsData;
import org.apache.hadoop.hive.metastore.api.Timestamp;
import org.apache.hadoop.hive.metastore.api.TimestampColumnStatsData;
import org.apache.hadoop.hive.ql.plan.ColStatistics;
import org.apache.hadoop.hive.ql.plan.ColStatistics.Range;
import org.apache.hadoop.hive.ql.plan.Statistics;
Expand Down Expand Up @@ -499,4 +503,66 @@ void testScaleColStatisticsPreservesUnknownNumFalses() {
assertEquals(-1, colStats.get(0).getNumFalses(), "Unknown numFalses (-1) should be preserved after scaling");
}

@Test
void testGetColStatisticsDateType() {
ColumnStatisticsObj cso = new ColumnStatisticsObj();
cso.setColName("date_col");
cso.setColType(serdeConstants.DATE_TYPE_NAME);

byte[] bitVectors = new byte[] {1, 2, 3, 4};
DateColumnStatsData dateStats = new DateColumnStatsData();
dateStats.setNumDVs(100);
dateStats.setNumNulls(10);
dateStats.setLowValue(new Date(18000)); // days since epoch
dateStats.setHighValue(new Date(19000));
dateStats.setBitVectors(bitVectors);

ColumnStatisticsData data = new ColumnStatisticsData();
data.setDateStats(dateStats);
cso.setStatsData(data);

ColStatistics cs = StatsUtils.getColStatistics(cso, "date_col");

assertNotNull(cs, "ColStatistics should not be null");
assertEquals(100, cs.getCountDistint(), "NumDVs mismatch for DATE");
assertEquals(10, cs.getNumNulls(), "NumNulls mismatch for DATE");
assertNotNull(cs.getBitVectors(), "BitVectors should not be null for DATE");

ColStatistics.Range range = cs.getRange();
assertNotNull(range, "Range should be created for DATE");
assertEquals(18000L, range.minValue.longValue(), "minValue mismatch for DATE");
assertEquals(19000L, range.maxValue.longValue(), "maxValue mismatch for DATE");
}

@Test
void testGetColStatisticsTimestampType() {
ColumnStatisticsObj cso = new ColumnStatisticsObj();
cso.setColName("ts_col");
cso.setColType(serdeConstants.TIMESTAMP_TYPE_NAME);

byte[] bitVectors = new byte[] {5, 6, 7, 8};
TimestampColumnStatsData tsStats = new TimestampColumnStatsData();
tsStats.setNumDVs(200);
tsStats.setNumNulls(20);
tsStats.setLowValue(new Timestamp(1600000000L));
tsStats.setHighValue(new Timestamp(1700000000L));
tsStats.setBitVectors(bitVectors);

ColumnStatisticsData data = new ColumnStatisticsData();
data.setTimestampStats(tsStats);
cso.setStatsData(data);

ColStatistics cs = StatsUtils.getColStatistics(cso, "ts_col");

assertNotNull(cs, "ColStatistics should not be null");
assertEquals(200, cs.getCountDistint(), "NumDVs mismatch for TIMESTAMP");
assertEquals(20, cs.getNumNulls(), "NumNulls mismatch for TIMESTAMP");
assertNotNull(cs.getBitVectors(), "BitVectors should not be null for TIMESTAMP");

ColStatistics.Range range = cs.getRange();
assertNotNull(range, "Range should be created for TIMESTAMP");
assertEquals(1600000000L, range.minValue.longValue(), "minValue mismatch for TIMESTAMP");
assertEquals(1700000000L, range.maxValue.longValue(), "maxValue mismatch for TIMESTAMP");
}

}
Loading
Loading