[common] Keep blank variant object keys unshredded during schema inference - #9813
zhuxiangyi wants to merge 1 commit into
Conversation
JingsongLi
left a comment
There was a problem hiding this comment.
Requirement fit: SUPPORTED. Implementation: CLEAN.
Reviewed 5824426c8d13. A valid Variant object containing a blank key must remain writable when inferred shredding is enabled. Leaving only those keys in the unshredded value preserves the payload and permits the other fields to be shredded. The adjusted ordering loop correctly uses the retained fields.
Validation: 43 tests passed across InferVariantShreddingSchemaTest and InferVariantShreddingWriteTest, using changed-source compilation against cached dependencies. Current head CI is green; Spark tests were inspected but not rerun locally.
No actionable implementation regression found in this review.
|
Please resolve conflicts. |
…rence
InferVariantShreddingSchema turns every key of a sampled object into a
RowType field, and RowType rejects an empty or whitespace-only field name.
A variant like {"":1,"a":2} is valid, but with
variant.inferShreddingSchema enabled the whole file failed to write with
"Field names must contain at least one non-whitespace character".
Skip such keys when building the inferred object type so they stay in the
unshredded value; the other keys are still shredded, and an object made
only of blank keys stays unshredded.
5824426 to
dfd6bad
Compare
|
Thanks for the review. Rebased onto master after #9811 landed (the two PRs added tests at the same spots in |
Purpose
This is a bug fix. With
variant.inferShreddingSchema=true, a Variant object whose key is empty or whitespace-only fails the whole file write:InferVariantShreddingSchema.schemaOfturns every key of a sampled object into aDataFieldand builds aRowTypefrom them, andRowTyperejects a blank field name. An empty key is a valid Variant object key (the same row writes and reads back fine with inference off), so inference must not turn it into a write failure. Other unusual keys (a.b, keys with spaces,typed_value/value/metadata, non-ASCII) already work.The fix skips blank keys when building the inferred object type, so they stay in the unshredded
valuewhile the remaining keys are still shredded; an object made only of blank keys stays unshredded. The alphabetical-order check now iterates the kept fields rather than the original object size.Tests
The new tests reproduce the bug: on
masterthey fail with theIllegalArgumentExceptionabove, and pass with this change.InferVariantShreddingSchemaTest#testInferSchemaSkipsBlankKeys:{"":1," ":2,"a":3}infers to a typed object with onlya;{"":1}alone stays unshredded.InferVariantShreddingWriteTest#testInferSchemaSkipsBlankKeys: writes such rows with inference on, checks the physical Parquet schema, the reconstructed Variant (blank keys preserved) and typed extraction of$.a.VariantTestBase: inserts{"":1,"a":2}and reads it back withto_jsonandvariant_get; runs under all four Spark 4.x configurations. Verified on Spark 4.1.2: 116 tests pass.API and Format
No.
Documentation
No.