Skip to content

Add opt-in exact decimal CSV schema inference - #118

Merged
osipovartem merged 1 commit into
df55-list-scalar-field-metadatafrom
df55-csv-decimal-inference
Oct 8, 2026
Merged

osipovartem merged 1 commit into
df55-list-scalar-field-metadatafrom
df55-csv-decimal-inference

Conversation

@osipovartem

Copy link
Copy Markdown
Collaborator

Summary

  • Add an opt-in CsvFormat::infer_schema_with_decimal method for one CSV object; the existing FileFormat inference path is unchanged.
  • Merge exact Decimal128 precision/scale across streaming chunks, using Arrow's chunk-local zero-integral metadata to avoid spurious Float64 at scale 38.
  • Keep mixed numeric/text inference deterministic, and retain a bounded, single-pass stream over the object.
  • Pin Arrow to the merged Embucket CSV decimal inference changes.

Validation

  • cargo +1.95.0 test -p datafusion-datasource-csv --lib --quiet: 10/10 passed, including multi-chunk decimal, scientific notation, text, zero-only, and 38-digit scale boundaries.
  • cargo +1.95.0 clippy -p datafusion-datasource-csv --all-targets --quiet: passed; only a pre-existing warning in datafusion-common remains.
  • cargo +1.95.0 fmt -p datafusion-datasource-csv and git diff --check: passed.
  • Independent read-only review: approved after addressing two correctness findings.

The opt-in path does not add row-level work to the default CSV inference or scan paths.

@osipovartem
osipovartem merged commit 462f07c into df55-list-scalar-field-metadata Oct 8, 2026
17 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant