Skip to content

Added iceberg write configs#11776

Open
infvg wants to merge 1 commit intoapache:mainfrom
infvg:icebergwriteconfigs
Open

Added iceberg write configs#11776
infvg wants to merge 1 commit intoapache:mainfrom
infvg:icebergwriteconfigs

Conversation

@infvg
Copy link
Copy Markdown
Contributor

@infvg infvg commented Mar 17, 2026

Adds support for: write.target-file-size-bytes, write.parquet.page-size-bytes
Resolves part of #11703

@weiting-chen weiting-chen mentioned this pull request Mar 25, 2026
60 tasks
@infvg infvg force-pushed the icebergwriteconfigs branch 2 times, most recently from 96c80e3 to a4db501 Compare March 31, 2026 07:10
# Conflicts:
#	docs/velox-configuration.md
@infvg infvg force-pushed the icebergwriteconfigs branch from a4db501 to 5bf8aea Compare March 31, 2026 07:19
@infvg infvg marked this pull request as ready for review March 31, 2026 07:36
@rui-mo rui-mo requested a review from jinchengchenghh April 2, 2026 08:58
Copy link
Copy Markdown
Contributor

@rui-mo rui-mo left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good to me. I’ll also loop in @jinchengchenghh for a final pass.

.booleanConf
.createWithDefault(true)

val PARQUET_PAGE_SIZE_BYTES =
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please remove the config, the config should not exposed to users, we should respect iceberg corresponding config, in this case, please update the config map as https://github.com/apache/gluten/blob/main/backends-velox/src-iceberg/main/scala/org/apache/gluten/execution/AbstractIcebergWriteExec.scala#L37

Please find the codec path, and set to Velox config.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants