Support Arrow View types in bulk copy

[gargsaumya]

Description

Adds native Arrow Utf8View and BinaryView support to mssql-py-core bulk copy. String views now map to SQL character, GUID, XML, and JSON destinations, while binary views map to binary destinations without requiring producers such as Polars to re-encode their buffers.

Includes Rust unit coverage, live PyArrow C-stream integration tests, and an unreleased changelog entry. The existing mssql-py-core version remains 0.1.9, which is reserved on main and is not yet published to the public wheel feed.

Related Issues

microsoft/mssql-python#708
https://sqlclientdrivers.visualstudio.com/mssql-python/_workitems/edit/47279

Validation

• cargo nextest run --lib arrow_bulkcopy::tests (53 passed)
• cargo clippy --frozen --all-features --all-targets -- -D warnings
• cargo fmt -- --check
• Live SQL Server tests for string_view and binary_view (2 passed)
• Live Polars 1.39.3 reproduction through mssql-python

Checklist

• cargo bfmt passes for the modified crate
• cargo bclippy passes for the modified crate
• cargo btest passes for the full workspace
• New/changed functionality has tests
• Public behavior changes are documented

[Copilot]

Pull request overview

Adds Arrow view-type support to mssql-py-core bulk copy, resolving Polars compatibility issue #708.

Changes:

• Maps Utf8View and BinaryView to compatible SQL types.
• Adds Rust unit and live integration coverage.
• Documents the change.

Reviewed changes

Copilot reviewed 4 out of 4 changed files in this pull request and generated 1 comment.

File	Description
mssql-py-core/src/arrow_bulkcopy.rs	Implements view-type planning and extraction.
mssql-py-core/tests/test_bulkcopy_arrow_varchar.py	Tests string_view round-tripping.
mssql-py-core/tests/test_bulkcopy_arrow_binary.py	Tests binary_view round-tripping.
CHANGELOG.md	Adds an unreleased changelog entry.

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

[Copilot]

Pull request overview

Copilot reviewed 4 out of 4 changed files in this pull request and generated no new comments.

[David Engel (David-Engel)]

Summary

Adds Arrow Utf8View/BinaryView support to mssql-py-core's Arrow bulk copy so Polars-exported C-stream columns load without a PyArrow re-encode. The change is small, correct, and well covered. No blocking findings — a couple of suggestions and nits inline.

Verified locally (worktree at 496d0c8e)

• cargo fmt -- --check — clean
• cargo clippy --all-features --all-targets -- -D warnings (mssql-py-core) — clean
• cargo nextest run --lib arrow_bulkcopy — 53 passed
• CI build 166711 Linux job: both new integration tests PASSED (test_cursor_bulkcopy_arrow_varbinary_binary_view, test_cursor_bulkcopy_arrow_varchar_string_view) — confirmed not skipped
• arrow 55.2 with features = ["ffi"] does support view types over the C data interface, and CI installs unpinned pyarrow, so pa.string_view() / pa.binary_view() are available

Correctness

The new resolve_kind arms cover NVarChar/NChar/NText, VarChar/Char/Text, VarBinary/Binary/Image, plus Xml/Json/UniqueIdentifier for Utf8View — mirroring the existing Utf8/LargeUtf8/Binary arms with no gaps I could find. NULL handling sits upstream of the match in extract_value, so view arrays get it for free, and the tests exercise both the inline (<=12 byte) and out-of-line buffer paths on each new array type.

CI

Both red checks trace to one infra flake, not this change:

• Build Stage / Build Linux ARM failed at Wait for cross-pool SQL host endpoint — "No SQL host endpoint after 10 min on stage attempt 3."
• coverage-report then timed out after 75 minutes waiting for the combined coverage artifact that the failed stage never produced.

The pipeline's own message applies here: use Rerun stage, not Rerun failed jobs — the latter won't restart the SQL host job and will time out again.

[github-actions]

📊 Code Coverage Report

🔥 Diff Coverage 89%

🎯 Overall Coverage 91.5%

📦 Project: mssql-tds + mssql-odbc + mssql-py-core ℹ️ Note: diff coverage is reported, not enforced.

---

Diff Coverage

Diff: main...HEAD, staged and unstaged changes

• mssql-py-core/src/arrow_bulkcopy.rs (89.9%): Missing lines 683-686,1195,1199,1226

Summary

• Total: 69 lines
• Missing: 7 lines
• Coverage: 89%

mssql-py-core/src/arrow_bulkcopy.rs

  679         Ok(a.value(row_idx))
  680     } else if let Some(a) = arr.as_any().downcast_ref::<StringViewArray>() {
  681         Ok(a.value(row_idx))
  682     } else {
! 683         Err(Error::UsageError(format!(
! 684             "Arrow array downcast failed: expected a UTF-8 string array (utf8/large_utf8/utf8_view), got {:?}",
! 685             arr.data_type()
! 686         )))
  687     }
  688 }
  689 
  690 /// Read a timestamp value as 100-ns ticks since UNIX epoch.

---

  1191             let dest = meta("name", sql_type, true);
  1192             let plan = one_col_plan(DataType::Utf8View, &dest);
  1193             match plan.extract_value(array.as_ref(), 0, &dest).unwrap() {
  1194                 ColumnValues::String(value) => assert_eq!(value.to_utf8_string(), "inline"),
! 1195                 other => panic!("expected String, got {other:?}"),
  1196             }
  1197             match plan.extract_value(array.as_ref(), 1, &dest).unwrap() {
  1198                 ColumnValues::String(value) => assert_eq!(value.to_utf8_string(), external),
! 1199                 other => panic!("expected String, got {other:?}"),
  1200             }
  1201             assert_eq!(
  1202                 plan.extract_value(array.as_ref(), 2, &dest).unwrap(),
  1203                 ColumnValues::Null

---

  1222                 .extract_value(array.as_ref(), 0, &dest)
  1223                 .unwrap();
  1224             match (sql_type, value) {
  1225                 (SqlDbType::Xml, ColumnValues::Xml(value)) => {
! 1226                     assert_eq!(value.as_string(), text)
  1227                 }
  1228                 (SqlDbType::Json, ColumnValues::Json(value)) => {
  1229                     assert_eq!(value.bytes, text.as_bytes())
  1230                 }

---

---

🔗 Quick Links

View Azure DevOps Build · Coverage Report

[Saurabh Singh (saurabh500)]

Adds native Arrow Utf8View and BinaryView support to Python bulk copy so Polars C-stream inputs no longer need a PyArrow table conversion. The implementation is correct and well scoped: destination mappings mirror the existing UTF-8 and binary paths, null handling is preserved, and tests cover inline and external view storage.

Verdict: Approve with comments. No blocking findings. One non-blocking unit-test assertion should be tightened; see the inline suggestion.

[Saurabh Singh (saurabh500)]

One minor test change, ready to approve after that is incorporated

[Saurabh Singh (saurabh500)]

gargsaumya can we drive this to completion please

[Saurabh Singh (saurabh500)]

Re-review — Tighten Utf8View XML/JSON test assertions (18855fd)

Re-checked the single 🔵 follow-up I left on my approval. Addressed verbatim: the Utf8View test now binds the extracted value and matches (sql_type, value) together — asserting Xml → as_string() == text and Json → bytes == text.as_bytes(), with a (_, other) => panic! catch-all. It no longer accepts either variant for either iteration and it checks the payload rather than just the discriminant, so it would fail if the planner swapped the Utf8View → Xml/Json mappings. CI is green on the head commit.

No further comments. Verdict: Approve — the substance was already approved; this closes the only note I left.