Skip to content

Pin Arrow CSV blank-row and source-position integration - #148

Merged
osipovartem merged 8 commits into
df55-csv-physical-header-skipfrom
df55-csv-blank-source-position
Oct 13, 2026
Merged

osipovartem merged 8 commits into
df55-csv-physical-header-skipfrom
df55-csv-blank-source-position

Conversation

@osipovartem

@osipovartem osipovartem commented Oct 13, 2026 •

Copy link
Copy Markdown
Collaborator

Summary

  • Pass the reviewed Arrow CSV blank-record policies and per-file Skip counters through DataFusion CSV scans.
  • Preserve intra-file repartitioning for strict policies when ranges are safe (no handler, physical-header skip, or multiline values). On a later-range decoder error only, replay the same file once to recover file-relative diagnostics; use ETag If-Match when available and keep replay out of Skip counters.
  • Reject protobuf serialization if it would silently discard policy or counter state. Keep the default Ignore path unchanged.
  • Rustice #661 sets newlines_in_values for quoted COPY input so quoted multiline records remain unsplit.

Verification

  • DataFusion CSV library: 28 passed, 1 ignored benchmark; includes LF/CRLF splits, EmitNullRow, per-file Skip counters, and replaced-object replay.
  • CSV crate Clippy: passed with no warnings in the changed crate (dependency warnings remain).
  • Ignored 64 MB in-memory scan benchmark: 61.8 MiB/s serial vs 420.6 MiB/s over eight ranges (median of three). This is scan-level, not end-to-end COPY throughput.
  • Rustice #661 repinned to 02d5697: targeted COPY blank-line fixture 55/55; full bronze corpus 3217/3217 with zero baseline regressions.
  • Independent read-only review: approved with no blocking correctness, performance, or API findings.

@osipovartem
osipovartem merged commit f0a25c6 into df55-csv-physical-header-skip Oct 13, 2026
31 of 36 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant