Skip to content

[GLUTEN-6887][VL] Daily Update Velox Version (2026_04_01) #11860

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
zhouyuan merged 15 commits into from
Apr 3, 2026
Merged

[GLUTEN-6887][VL] Daily Update Velox Version (2026_04_01) #11860

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
15 commits
Select commit Hold shift + click to select a range
fe79614
[GLUTEN-6887][VL] Daily Update Velox Version (dft-2026_04_01)
Apr 1, 2026
36bf7c3
Fix SPARK-18108: exclude partition columns from HiveTableHandle dataC…
baibaichen Mar 3, 2026
4c29b7e
Point Velox to PR3 branch with parquet type widening support
baibaichen Mar 6, 2026
5870178
Update VeloxTestSettings for Velox PR2
baibaichen Mar 3, 2026
b0c53d3
Disable native writer for ParquetTypeWideningSuite
baibaichen Mar 7, 2026
f590514
Override 33 type widening tests with expectError=true
baibaichen Mar 11, 2026
48dd3d8
fix velox rebase
zhouyuan Apr 1, 2026
ac91ab2
ignore ut
zhouyuan Apr 1, 2026
75e6871
ignore more ut
zhouyuan Apr 1, 2026
e7f9ba4
fix ignore api
zhouyuan Apr 1, 2026
4d99fd3
ignore failed ut
zhouyuan Apr 2, 2026
df9b816
fix on clickhouse tpcds queries
zhouyuan Apr 2, 2026
27e83cf
fix q30
zhouyuan Apr 2, 2026
7384ed2
ignore ut
zhouyuan Apr 2, 2026
e4499a6
fix
zhouyuan Apr 3, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
26 changes: 13 additions & 13 deletions .github/workflows/docker_image.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -44,7 +44,7 @@ jobs:
uses: docker/setup-buildx-action@8d2750c68a42422c14e847fe6c8ac0403b4cbd6f

- name: Login to Docker Hub
uses: docker/login-action@c94ce9fb468520275223c153574b00df6fe4bcc9@v2
uses: docker/login-action@c94ce9fb468520275223c153574b00df6fe4bcc9
with:
username: ${{ secrets.DOCKERHUB_USER }}
password: ${{ secrets.DOCKERHUB_TOKEN }}
Expand All @@ -69,7 +69,7 @@ jobs:
uses: docker/setup-buildx-action@8d2750c68a42422c14e847fe6c8ac0403b4cbd6f

- name: Login to Docker Hub
uses: docker/login-action@c94ce9fb468520275223c153574b00df6fe4bcc9@v2
uses: docker/login-action@c94ce9fb468520275223c153574b00df6fe4bcc9
with:
username: ${{ secrets.DOCKERHUB_USER }}
password: ${{ secrets.DOCKERHUB_TOKEN }}
Expand Down Expand Up @@ -100,7 +100,7 @@ jobs:
uses: docker/setup-buildx-action@8d2750c68a42422c14e847fe6c8ac0403b4cbd6f

- name: Login to Docker Hub
uses: docker/login-action@c94ce9fb468520275223c153574b00df6fe4bcc9@v2
uses: docker/login-action@c94ce9fb468520275223c153574b00df6fe4bcc9
with:
username: ${{ secrets.DOCKERHUB_USER }}
password: ${{ secrets.DOCKERHUB_TOKEN }}
Expand All @@ -125,7 +125,7 @@ jobs:
uses: docker/setup-buildx-action@8d2750c68a42422c14e847fe6c8ac0403b4cbd6f

- name: Login to Docker Hub
uses: docker/login-action@c94ce9fb468520275223c153574b00df6fe4bcc9@v2
uses: docker/login-action@c94ce9fb468520275223c153574b00df6fe4bcc9
with:
username: ${{ secrets.DOCKERHUB_USER }}
password: ${{ secrets.DOCKERHUB_TOKEN }}
Expand Down Expand Up @@ -155,7 +155,7 @@ jobs:
uses: docker/setup-buildx-action@8d2750c68a42422c14e847fe6c8ac0403b4cbd6f

- name: Login to Docker Hub
uses: docker/login-action@c94ce9fb468520275223c153574b00df6fe4bcc9@v2
uses: docker/login-action@c94ce9fb468520275223c153574b00df6fe4bcc9
with:
username: ${{ secrets.DOCKERHUB_USER }}
password: ${{ secrets.DOCKERHUB_TOKEN }}
Expand Down Expand Up @@ -191,7 +191,7 @@ jobs:
uses: docker/setup-buildx-action@8d2750c68a42422c14e847fe6c8ac0403b4cbd6f

- name: Login to Docker Hub
uses: docker/login-action@c94ce9fb468520275223c153574b00df6fe4bcc9@v2
uses: docker/login-action@c94ce9fb468520275223c153574b00df6fe4bcc9
with:
username: ${{ secrets.DOCKERHUB_USER }}
password: ${{ secrets.DOCKERHUB_TOKEN }}
Expand Down Expand Up @@ -241,7 +241,7 @@ jobs:
uses: docker/setup-buildx-action@8d2750c68a42422c14e847fe6c8ac0403b4cbd6f

- name: Login to Docker Hub
uses: docker/login-action@c94ce9fb468520275223c153574b00df6fe4bcc9@v2
uses: docker/login-action@c94ce9fb468520275223c153574b00df6fe4bcc9
with:
username: ${{ secrets.DOCKERHUB_USER }}
password: ${{ secrets.DOCKERHUB_TOKEN }}
Expand Down Expand Up @@ -291,7 +291,7 @@ jobs:
uses: docker/setup-buildx-action@8d2750c68a42422c14e847fe6c8ac0403b4cbd6f

- name: Login to Docker Hub
uses: docker/login-action@c94ce9fb468520275223c153574b00df6fe4bcc9@v2
uses: docker/login-action@c94ce9fb468520275223c153574b00df6fe4bcc9
with:
username: ${{ secrets.DOCKERHUB_USER }}
password: ${{ secrets.DOCKERHUB_TOKEN }}
Expand Down Expand Up @@ -353,7 +353,7 @@ jobs:
uses: docker/setup-buildx-action@8d2750c68a42422c14e847fe6c8ac0403b4cbd6f

- name: Login to Docker Hub
uses: docker/login-action@c94ce9fb468520275223c153574b00df6fe4bcc9@v2
uses: docker/login-action@c94ce9fb468520275223c153574b00df6fe4bcc9
with:
username: ${{ secrets.DOCKERHUB_USER }}
password: ${{ secrets.DOCKERHUB_TOKEN }}
Expand Down Expand Up @@ -416,7 +416,7 @@ jobs:
uses: docker/setup-buildx-action@8d2750c68a42422c14e847fe6c8ac0403b4cbd6f

- name: Login to Docker Hub
uses: docker/login-action@c94ce9fb468520275223c153574b00df6fe4bcc9@v2
uses: docker/login-action@c94ce9fb468520275223c153574b00df6fe4bcc9
with:
username: ${{ secrets.DOCKERHUB_USER }}
password: ${{ secrets.DOCKERHUB_TOKEN }}
Expand Down Expand Up @@ -464,7 +464,7 @@ jobs:
merge-multiple: true

- name: Login to Docker Hub
uses: docker/login-action@c94ce9fb468520275223c153574b00df6fe4bcc9@v2
uses: docker/login-action@c94ce9fb468520275223c153574b00df6fe4bcc9
with:
username: ${{ secrets.DOCKERHUB_USER }}
password: ${{ secrets.DOCKERHUB_TOKEN }}
Expand Down Expand Up @@ -506,7 +506,7 @@ jobs:
merge-multiple: true

- name: Login to Docker Hub
uses: docker/login-action@c94ce9fb468520275223c153574b00df6fe4bcc9@v2
uses: docker/login-action@c94ce9fb468520275223c153574b00df6fe4bcc9
with:
username: ${{ secrets.DOCKERHUB_USER }}
password: ${{ secrets.DOCKERHUB_TOKEN }}
Expand Down Expand Up @@ -549,7 +549,7 @@ jobs:
merge-multiple: true

- name: Login to Docker Hub
uses: docker/login-action@c94ce9fb468520275223c153574b00df6fe4bcc9@v2
uses: docker/login-action@c94ce9fb468520275223c153574b00df6fe4bcc9
with:
username: ${{ secrets.DOCKERHUB_USER }}
password: ${{ secrets.DOCKERHUB_TOKEN }}
Expand Down
3 changes: 2 additions & 1 deletion ...house/src/test/scala/org/apache/gluten/execution/GlutenClickHouseTPCDSAbstractSuite.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -43,8 +43,9 @@ abstract class GlutenClickHouseTPCDSAbstractSuite

protected val tablesPath: String = UTSystemParameters.tpcdsDecimalDataPath + "/"
protected val db_name: String = "tpcdsdb"
// TODO: fix to use the new DS queries https://github.com/apache/gluten/issues/11871
protected val tpcdsQueries: String =
Copy link
Copy Markdown
Member

@zhouyuan zhouyuan Apr 2, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@zzcclp

resPath + "../../../../tools/gluten-it/common/src/main/resources/tpcds-queries"
resPath + "../../../../tools/gluten-it/common/src/main/resources/tpcds-queries-clickhouse"
protected val queriesResults: String = resPath + "tpcds-decimal-queries-output"

/** Return values: (sql num, is fall back) */
Expand Down
2 changes: 1 addition & 1 deletion ...ends-clickhouse/src/test/scala/org/apache/gluten/execution/GlutenEliminateJoinSuite.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -133,7 +133,7 @@ class GlutenEliminateJoinSuite extends GlutenClickHouseWholeStageTransformerSuit
})
}

test("Eliminate two aggregate joins with attribute reordered") {
ignore("Eliminate two aggregate joins with attribute reordered") {
val sql = """
Copy link
Copy Markdown
Member

@zhouyuan zhouyuan Apr 3, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@zzcclp this test failed, it's not related with this patch, seems due to the recent changes in the past two weeks

Copy link
Copy Markdown
Contributor

@zzcclp zzcclp Apr 3, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I will take a look next week.

select t1.k1, t1.k3, t2.k1, t2.k3, s1, s2 from (
select k1, k3, sum(v1) s1 from (
Expand Down
1 change: 1 addition & 0 deletions cpp/velox/config/VeloxConfig.h
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -165,6 +165,7 @@ const std::string kMemoryPoolCapacityTransferAcrossTasks =
"spark.gluten.sql.columnar.backend.velox.memoryPoolCapacityTransferAcrossTasks";
const std::string kOrcUseColumnNames = "spark.gluten.sql.columnar.backend.velox.orcUseColumnNames";
const std::string kParquetUseColumnNames = "spark.gluten.sql.columnar.backend.velox.parquetUseColumnNames";
const std::string kAllowInt32Narrowing = "spark.gluten.sql.columnar.backend.velox.allowInt32Narrowing";

// write fies
const std::string kMaxPartitions = "spark.gluten.sql.columnar.backend.velox.maxPartitionsPerWritersSession";
Expand Down
27 changes: 26 additions & 1 deletion cpp/velox/substrait/SubstraitToVeloxPlan.cc
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1498,6 +1498,31 @@ core::PlanNodePtr SubstraitToVeloxPlanConverter::toVeloxPlan(const ::substrait::
// The columns present in the table, if not available default to the baseSchema.
auto tableSchema = splitInfo->tableSchema ? splitInfo->tableSchema : baseSchema;

// Build dataColumns from tableSchema, excluding partition columns.
// HiveTableHandle::dataColumns() is used as fileSchema for the reader.
// Partition columns should not be validated against the file's physical types
// (their values come from the partition path, not from the file).
std::unordered_set<std::string> partitionColNames;
for (int idx = 0; idx < colNameList.size(); idx++) {
if (columnTypes[idx] == ColumnType::kPartitionKey) {
partitionColNames.insert(colNameList[idx]);
}
}
RowTypePtr dataColumns;
if (partitionColNames.empty()) {
dataColumns = tableSchema;
} else {
std::vector<std::string> dataColNames;
std::vector<TypePtr> dataColTypes;
for (int idx = 0; idx < tableSchema->size(); idx++) {
if (partitionColNames.find(tableSchema->nameOf(idx)) == partitionColNames.end()) {
dataColNames.push_back(tableSchema->nameOf(idx));
dataColTypes.push_back(tableSchema->childAt(idx));
}
}
dataColumns = ROW(std::move(dataColNames), std::move(dataColTypes));
}

connector::ConnectorTableHandlePtr tableHandle;
auto remainingFilter = readRel.has_filter() ? exprConverter_->toVeloxExpr(readRel.filter(), baseSchema) : nullptr;
auto connectorId = kHiveConnectorId;
Expand All @@ -1509,7 +1534,7 @@ core::PlanNodePtr SubstraitToVeloxPlanConverter::toVeloxPlan(const ::substrait::
}
common::SubfieldFilters subfieldFilters;
tableHandle = std::make_shared<connector::hive::HiveTableHandle>(
connectorId, "hive_table", std::move(subfieldFilters), remainingFilter, tableSchema);
connectorId, "hive_table", std::move(subfieldFilters), remainingFilter, dataColumns);

// Get assignments and out names.
std::vector<std::string> outNames;
Expand Down
2 changes: 2 additions & 0 deletions cpp/velox/utils/ConfigExtractor.cc
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -239,6 +239,8 @@ std::shared_ptr<facebook::velox::config::ConfigBase> createHiveConnectorSessionC
conf->get<bool>(kIgnoreMissingFiles, false) ? "true" : "false";
configs[facebook::velox::connector::hive::HiveConfig::kParquetUseColumnNamesSession] =
conf->get<bool>(kParquetUseColumnNames, true) ? "true" : "false";
configs[facebook::velox::connector::hive::HiveConfig::kAllowInt32NarrowingSession] =
conf->get<bool>(kAllowInt32Narrowing, true) ? "true" : "false";
configs[facebook::velox::connector::hive::HiveConfig::kOrcUseColumnNamesSession] =
conf->get<bool>(kOrcUseColumnNames, true) ? "true" : "false";

Expand Down
4 changes: 2 additions & 2 deletions ep/build-velox/src/get-velox.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -18,8 +18,8 @@ set -exu

CURRENT_DIR=$(cd "$(dirname "$BASH_SOURCE")"; pwd)
VELOX_REPO=https://github.com/IBM/velox.git
VELOX_BRANCH=dft-2026_03_24
VELOX_ENHANCED_BRANCH=ibm-2026_03_24
VELOX_BRANCH=dft-2026_04_01-iceberg
VELOX_ENHANCED_BRANCH=ibm-2026_04_01-fix
VELOX_HOME=""
RUN_SETUP_SCRIPT=ON
ENABLE_ENHANCED_FEATURES=OFF
Expand Down
2 changes: 2 additions & 0 deletions gluten-ut/spark33/src/test/scala/org/apache/gluten/utils/velox/VeloxTestSettings.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -444,6 +444,8 @@ class VeloxTestSettings extends BackendTestSettings {
.exclude("SPARK-38825: in and notIn filters")
enableSuite[GlutenParquetInteroperabilitySuite]
.exclude("parquet timestamp conversion")
// TODO: https://github.com/apache/gluten/issues/11865
.exclude("SPARK-36803: parquet files with legacy mode and schema evolution")
enableSuite[GlutenParquetIOSuite]
// Exception.
.exclude("SPARK-35640: read binary as timestamp should throw schema incompatible error")
Expand Down
3 changes: 2 additions & 1 deletion ...pache/spark/sql/execution/datasources/parquet/GlutenParquetThriftCompatibilitySuite.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -26,7 +26,8 @@ class GlutenParquetThriftCompatibilitySuite
getWorkspaceFilePath("sql", "core", "src", "test", "resources").toString +
"/test-data/parquet-thrift-compat.snappy.parquet"

testGluten("Read Parquet file generated by parquet-thrift") {
// TODO: https://github.com/apache/gluten/issues/11865
Copy link
Copy Markdown
Member

@zhouyuan zhouyuan Apr 1, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@baibaichen seems due to missing fix from one old OAP patch: https://github.com/IBM/velox/pull/35/changes

Copy link
Copy Markdown
Contributor

@baibaichen baibaichen Apr 2, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

it's in baibaichen/velox@9f58f05

Copy link
Copy Markdown
Member

@zhouyuan zhouyuan Apr 2, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

waiting for facebookincubator/velox#16966

ignoreGluten("Read Parquet file generated by parquet-thrift") {
logInfo(s"""Schema of the Parquet file written by parquet-thrift:
|${readParquetSchema(parquetFilePath.toString)}
""".stripMargin)
Expand Down
3 changes: 2 additions & 1 deletion ...pache/spark/sql/execution/datasources/parquet/GlutenParquetThriftCompatibilitySuite.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -26,7 +26,8 @@ class GlutenParquetThriftCompatibilitySuite
getWorkspaceFilePath("sql", "core", "src", "test", "resources").toString +
"/test-data/parquet-thrift-compat.snappy.parquet"

testGluten("Read Parquet file generated by parquet-thrift") {
// TODO: https://github.com/apache/gluten/issues/11865
ignoreGluten("Read Parquet file generated by parquet-thrift") {
logInfo(s"""Schema of the Parquet file written by parquet-thrift:
|${readParquetSchema(parquetFilePath.toString)}
""".stripMargin)
Expand Down
3 changes: 2 additions & 1 deletion ...pache/spark/sql/execution/datasources/parquet/GlutenParquetThriftCompatibilitySuite.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -26,7 +26,8 @@ class GlutenParquetThriftCompatibilitySuite
getWorkspaceFilePath("sql", "core", "src", "test", "resources").toString +
"/test-data/parquet-thrift-compat.snappy.parquet"

testGluten("Read Parquet file generated by parquet-thrift") {
// TODO: https://github.com/apache/gluten/issues/11865
ignoreGluten("Read Parquet file generated by parquet-thrift") {
logInfo(s"""Schema of the Parquet file written by parquet-thrift:
|${readParquetSchema(parquetFilePath.toString)}
""".stripMargin)
Expand Down
29 changes: 4 additions & 25 deletions gluten-ut/spark40/src/test/scala/org/apache/gluten/utils/velox/VeloxTestSettings.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -326,13 +326,17 @@ class VeloxTestSettings extends BackendTestSettings {
enableSuite[GlutenJsonParsingOptionsSuite]
// Generated suites for org.apache.spark.sql.execution.datasources.parquet
enableSuite[GlutenParquetAvroCompatibilitySuite]
// TODO: https://github.com/apache/gluten/issues/11865
.exclude("various complex types")
enableSuite[GlutenParquetCommitterSuite]
enableSuite[GlutenParquetFieldIdSchemaSuite]
enableSuite[GlutenParquetTypeWideningSuite]
// Velox does not support DELTA_BYTE_ARRAY encoding for FIXED_LEN_BYTE_ARRAY decimals.
.exclude("parquet decimal precision change Decimal(20, 2) -> Decimal(22, 2)")
.exclude("parquet decimal precision and scale change Decimal(20, 7) -> Decimal(22, 5)")
.exclude("parquet decimal precision and scale change Decimal(20, 5) -> Decimal(22, 8)")
.exclude("parquet decimal precision and scale change Decimal(20, 2) -> Decimal(22, 4)")
// Velox native reader aligns with vectorized reader behavior, always rejecting incompatible decimal conversions.
.exclude("parquet decimal precision and scale change Decimal(10, 4) -> Decimal(12, 7)")
.exclude("parquet decimal precision and scale change Decimal(10, 6) -> Decimal(12, 4)")
.exclude("parquet decimal precision and scale change Decimal(10, 7) -> Decimal(5, 2)")
Expand All @@ -344,22 +348,12 @@ class VeloxTestSettings extends BackendTestSettings {
.exclude("parquet decimal precision and scale change Decimal(22, 5) -> Decimal(20, 7)")
.exclude("parquet decimal precision and scale change Decimal(5, 2) -> Decimal(6, 4)")
.exclude("parquet decimal precision and scale change Decimal(7, 4) -> Decimal(5, 2)")
.exclude("parquet decimal precision and scale change Decimal(10, 2) -> Decimal(12, 4)")
.exclude("parquet decimal precision and scale change Decimal(10, 2) -> Decimal(20, 12)")
.exclude("parquet decimal precision and scale change Decimal(5, 2) -> Decimal(10, 7)")
.exclude("parquet decimal precision and scale change Decimal(5, 2) -> Decimal(20, 17)")
.exclude("parquet decimal precision and scale change Decimal(5, 2) -> Decimal(7, 4)")
.exclude("parquet decimal precision change Decimal(10, 2) -> Decimal(5, 2)")
.exclude("parquet decimal precision change Decimal(12, 2) -> Decimal(10, 2)")
.exclude("parquet decimal precision change Decimal(20, 2) -> Decimal(10, 2)")
.exclude("parquet decimal precision change Decimal(20, 2) -> Decimal(5, 2)")
.exclude("parquet decimal precision change Decimal(22, 2) -> Decimal(20, 2)")
.exclude("parquet decimal precision change Decimal(7, 2) -> Decimal(5, 2)")
.exclude("parquet decimal precision change Decimal(10, 2) -> Decimal(12, 2)")
.exclude("parquet decimal precision change Decimal(10, 2) -> Decimal(20, 2)")
.exclude("parquet decimal precision change Decimal(5, 2) -> Decimal(10, 2)")
.exclude("parquet decimal precision change Decimal(5, 2) -> Decimal(20, 2)")
.exclude("parquet decimal precision change Decimal(5, 2) -> Decimal(7, 2)")
.exclude("parquet decimal type change Decimal(5, 2) -> Decimal(3, 2) overflows with parquet-mr")
.exclude("unsupported parquet conversion ByteType -> DecimalType(1,0)")
.exclude("unsupported parquet conversion ByteType -> DecimalType(2,0)")
Expand All @@ -369,29 +363,14 @@ class VeloxTestSettings extends BackendTestSettings {
.exclude("unsupported parquet conversion IntegerType -> DecimalType(10,1)")
.exclude("unsupported parquet conversion IntegerType -> DecimalType(5,0)")
.exclude("unsupported parquet conversion IntegerType -> DecimalType(9,0)")
.exclude("unsupported parquet conversion LongType -> DateType")
.exclude("unsupported parquet conversion LongType -> DecimalType(10,0)")
.exclude("unsupported parquet conversion LongType -> DecimalType(19,0)")
.exclude("unsupported parquet conversion LongType -> DecimalType(20,1)")
.exclude("unsupported parquet conversion LongType -> IntegerType")
.exclude("unsupported parquet conversion ShortType -> DecimalType(3,0)")
.exclude("unsupported parquet conversion ShortType -> DecimalType(4,0)")
.exclude("unsupported parquet conversion ShortType -> DecimalType(5,0)")
.exclude("unsupported parquet conversion ShortType -> DecimalType(5,1)")
.exclude("unsupported parquet conversion ShortType -> DecimalType(6,1)")
.exclude("parquet widening conversion ByteType -> DecimalType(11,1)")
.exclude("parquet widening conversion ByteType -> DecimalType(20,0)")
.exclude("parquet widening conversion IntegerType -> DecimalType(11,1)")
.exclude("parquet widening conversion IntegerType -> DecimalType(20,0)")
.exclude("parquet widening conversion IntegerType -> DecimalType(38,0)")
.exclude("parquet widening conversion IntegerType -> DoubleType")
.exclude("parquet widening conversion LongType -> DecimalType(20,0)")
.exclude("parquet widening conversion LongType -> DecimalType(21,1)")
.exclude("parquet widening conversion LongType -> DecimalType(38,0)")
.exclude("parquet widening conversion ShortType -> DecimalType(11,1)")
.exclude("parquet widening conversion ShortType -> DecimalType(20,0)")
.exclude("parquet widening conversion ShortType -> DecimalType(38,0)")
.exclude("parquet widening conversion ShortType -> DoubleType")
enableSuite[GlutenParquetVariantShreddingSuite]
// Generated suites for org.apache.spark.sql.execution.datasources.text
// TODO: 4.x enableSuite[GlutenWholeTextFileV1Suite] // 1 failure
Expand Down
3 changes: 2 additions & 1 deletion ...pache/spark/sql/execution/datasources/parquet/GlutenParquetThriftCompatibilitySuite.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -26,7 +26,8 @@ class GlutenParquetThriftCompatibilitySuite
getWorkspaceFilePath("sql", "core", "src", "test", "resources").toString +
"/test-data/parquet-thrift-compat.snappy.parquet"

testGluten("Read Parquet file generated by parquet-thrift") {
// TODO: https://github.com/apache/gluten/issues/11865
ignoreGluten("Read Parquet file generated by parquet-thrift") {
logInfo(s"""Schema of the Parquet file written by parquet-thrift:
|${readParquetSchema(parquetFilePath.toString)}
""".stripMargin)
Expand Down
Loading
Loading