Skip to content

[common] Keep blank variant object keys unshredded during schema inference - #9813

Open
zhuxiangyi wants to merge 1 commit into
apache:masterfrom
zhuxiangyi:variant-infer-blank-key
Open

zhuxiangyi wants to merge 1 commit into
apache:masterfrom
zhuxiangyi:variant-infer-blank-key

Conversation

@zhuxiangyi

Copy link
Copy Markdown
Contributor

Purpose

This is a bug fix. With variant.inferShreddingSchema=true, a Variant object whose key is empty or whitespace-only fails the whole file write:

CREATE TABLE T (id INT, v VARIANT) TBLPROPERTIES ('variant.inferShreddingSchema' = 'true');
INSERT INTO T VALUES (1, parse_json('{"":1,"a":2}'));
-- Job aborted ... java.lang.IllegalArgumentException: Field names must contain at least one non-whitespace character.
--   at org.apache.paimon.types.RowType.validateFields(RowType.java:283)
--   at org.apache.paimon.data.variant.InferVariantShreddingSchema.schemaOf(InferVariantShreddingSchema.java:318)

InferVariantShreddingSchema.schemaOf turns every key of a sampled object into a DataField and builds a RowType from them, and RowType rejects a blank field name. An empty key is a valid Variant object key (the same row writes and reads back fine with inference off), so inference must not turn it into a write failure. Other unusual keys (a.b, keys with spaces, typed_value / value / metadata, non-ASCII) already work.

The fix skips blank keys when building the inferred object type, so they stay in the unshredded value while the remaining keys are still shredded; an object made only of blank keys stays unshredded. The alphabetical-order check now iterates the kept fields rather than the original object size.

Tests

The new tests reproduce the bug: on master they fail with the IllegalArgumentException above, and pass with this change.

  • InferVariantShreddingSchemaTest#testInferSchemaSkipsBlankKeys: {"":1," ":2,"a":3} infers to a typed object with only a; {"":1} alone stays unshredded.
  • InferVariantShreddingWriteTest#testInferSchemaSkipsBlankKeys: writes such rows with inference on, checks the physical Parquet schema, the reconstructed Variant (blank keys preserved) and typed extraction of $.a.
  • VariantTestBase: inserts {"":1,"a":2} and reads it back with to_json and variant_get; runs under all four Spark 4.x configurations. Verified on Spark 4.1.2: 116 tests pass.

API and Format

No.

Documentation

No.

@JingsongLi JingsongLi left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Requirement fit: SUPPORTED. Implementation: CLEAN.

Reviewed 5824426c8d13. A valid Variant object containing a blank key must remain writable when inferred shredding is enabled. Leaving only those keys in the unshredded value preserves the payload and permits the other fields to be shredded. The adjusted ordering loop correctly uses the retained fields.

Validation: 43 tests passed across InferVariantShreddingSchemaTest and InferVariantShreddingWriteTest, using changed-source compilation against cached dependencies. Current head CI is green; Spark tests were inspected but not rerun locally.

No actionable implementation regression found in this review.

@JingsongLi

Copy link
Copy Markdown
Contributor

Please resolve conflicts.

…rence

InferVariantShreddingSchema turns every key of a sampled object into a
RowType field, and RowType rejects an empty or whitespace-only field name.
A variant like {"":1,"a":2} is valid, but with
variant.inferShreddingSchema enabled the whole file failed to write with
"Field names must contain at least one non-whitespace character".

Skip such keys when building the inferred object type so they stay in the
unshredded value; the other keys are still shredded, and an object made
only of blank keys stays unshredded.
@zhuxiangyi
zhuxiangyi force-pushed the variant-infer-blank-key branch from 5824426 to dfd6bad Compare September 15, 2026 03:45
@zhuxiangyi

Copy link
Copy Markdown
Contributor Author

Thanks for the review. Rebased onto master after #9811 landed (the two PRs added tests at the same spots in InferVariantShreddingSchemaTest, InferVariantShreddingWriteTest and VariantTestBase); conflicts are resolved and the change is still a single commit. After the rebase: InferVariantShreddingSchemaTest / InferVariantShreddingWriteTest pass locally, and the four Spark 4.1 Variant suites pass (124 tests). CI is re-running on the new head.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants