GH-41670: [C++][Python] Move to DLPack 1.3

[AntoinePrv]

Rationale for this change

Pure version bump to enable implementing more features.

What changes are included in this PR?

• Change the DLPack header file and definition.
• Use raw_data for Tensor similar to Array (instead of mutable_raw_data that is not available on immutable tensors).
• Strides may no longer be NULL
• Add ExportArrayVersioned and ExportTensorVersioned C APIs
• Support for copy
• Support for max_version and copy in __dlpack__

Are these changes tested?

Yes, with existing tests.

Are there any user-facing changes?

• GitHub Issue: [C++][Python] Update DLPack version #41670

[github-actions]

⚠️ GitHub issue #41670 has been automatically assigned in GitHub to PR creator.

[Copilot]

🟡 Changes recommended

Arrow’s current DLPack Array exporter still emits strides = NULL (non-compliant with DLPack v1.2+ requirements) and the new header text contains documentation mismatches that should be corrected.

Once you've addressed the issues Copilot identified, you can request another Copilot review.

This review doesn't count toward merge requirements. Sign up for the private preview to control whether Copilot approvals count.

Pull request overview

Updates Arrow’s vendored DLPack ABI header to DLPack v1.3, expanding the ABI surface (new device/type enums, flags, and the __dlpack_c_exchange_api__ protocol) while bumping the reported minor version.

Changes:

• Bump DLPack minor version to 1.3 and sync header contents to a newer upstream commit.
• Add new DLDeviceType/DLDataTypeCode values and additional DLManagedTensorVersioned flag bitmasks.
• Introduce DLPack fast exchange protocol (DLPackExchangeAPI*) type definitions and documentation.

File summaries

File	Description
cpp/src/arrow/c/dlpack_abi.h	Vendor update to DLPack 1.3 ABI definitions, enums, flags, and fast exchange protocol types.

Review details

Suppressed comments (1)

cpp/src/arrow/c/dlpack_abi.h:426

• The documentation for DLPackManagedTensorFromPyObjectNoSync claims the function returns an owning pointer/NULL, but the typedef returns int and delivers the tensor via out. The return contract should match the signature.

 * \return The owning DLManagedTensorVersioned* or NULL on failure with a
 *         Python exception set. If the data cannot be described using DLPack
 *         this should be a BufferError if possible.
 * \note - As a C function, must not thrown C++ exceptions.

• Files reviewed: 1/1 changed files
• Comments generated: 3
• Review effort level: Lite

We're testing this review assessment. Please use 👍 or 👎 to tell us if it's correct.

[Copilot]

🟡 Changes recommended

The new public versioned export APIs are not covered by tests, and there is a small but concrete maintainability issue (std::move on a const shared_ptr reference) to address.

Once you've addressed the issues Copilot identified, you can request another Copilot review.

This review doesn't count toward merge requirements. Sign up for the private preview to control whether Copilot approvals count.

Review details

Suppressed comments (3)

cpp/src/arrow/c/dlpack.cc:212

• t is a const std::shared_ptr<Tensor>&, so std::move(t) won’t actually move (it will fall back to a copy via operator=(const shared_ptr&)). Using a plain copy here is clearer and avoids the misleading std::move.

  ctx->t = std::move(t);

cpp/src/arrow/c/dlpack_test.cc:49

• New public APIs ExportArrayVersioned/ExportTensorVersioned (and the version/flags fields they populate) aren’t covered by tests here; currently the tests only exercise the legacy ExportArray/ExportTensor paths. Adding a small set of assertions for version/flags on the versioned exports would prevent regressions.

  ASSERT_EQ(1, *dltensor.strides);  // Must be non-null with ndim>0 since 1.2

cpp/src/arrow/c/dlpack.h:48

• ExportTensor also returns the legacy DLManagedTensor (deprecated in DLPack in favor of DLManagedTensorVersioned), but only ExportArray is annotated as deprecated in this header. To keep the API docs consistent and steer callers to the versioned path, add the same deprecation note for ExportTensor.

ARROW_EXPORT
Result<DLManagedTensor*> ExportTensor(const std::shared_ptr<Tensor>& t);

• Files reviewed: 4/4 changed files
• Comments generated: 0 new
• Review effort level: Lite

We're testing this review assessment. Please use 👍 or 👎 to tell us if it's correct.

[Copilot]

🟡 Changes recommended

cpp/src/arrow/c/dlpack.cc has missing semicolons after ARROW_ASSIGN_OR_RAISE statements that will break compilation.

Once you've addressed the issues Copilot identified, you can request another Copilot review.

This review doesn't count toward merge requirements. Sign up for the private preview to control whether Copilot approvals count.

Review details

Suppressed comments (1)

cpp/src/arrow/c/dlpack.cc:175

• Missing semicolon after ARROW_ASSIGN_OR_RAISE; as written this won’t compile because the macro expansion isn’t terminated as a statement.

  ARROW_ASSIGN_OR_RAISE(auto device, ExportDevice(t))

• Files reviewed: 4/4 changed files
• Comments generated: 1
• Review effort level: Lite

We're testing this review assessment. Please use 👍 or 👎 to tell us if it's correct.

[AntoinePrv]

@AlenkaF would you mind providing early feedback before I move on to the Python side.

[Copilot]

🟡 Changes recommended

The legacy ExportTensor path now exposes immutable tensor memory without a read-only signal, which can violate Arrow’s immutability contract unless behavior/tests are adjusted accordingly.

Once you've addressed the issues Copilot identified, you can request another Copilot review.

This review doesn't count toward merge requirements. Sign up for the private preview to control whether Copilot approvals count.

Review details

Suppressed comments (2)

cpp/src/arrow/c/dlpack.h:48

• ExportTensor is also a legacy (pre-versioned) DLPack API, but only ExportArray is documented as deprecated. This makes the header inconsistent and can mislead users into using the legacy tensor export instead of the versioned API.

ARROW_EXPORT
Result<DLManagedTensor*> ExportTensor(const std::shared_ptr<Tensor>& t);

cpp/src/arrow/c/dlpack_test.cc:265

• This test currently validates exporting an immutable tensor for both legacy and versioned producers. If the legacy API is made to reject immutable tensors (since it cannot express read-only), the test should branch: expect a TypeError for LegacyProducer and continue to validate successful export + read-only flag for VersionedProducer.

  CheckDLTensor<TypeParam>(read_only_tensor, float32(), DLDataTypeCode::kDLFloat, shape,
                           dlpack_strides);
}

• Files reviewed: 4/4 changed files
• Comments generated: 1
• Review effort level: Lite

We're testing this review assessment. Please use 👍 or 👎 to tell us if it's correct.

[Copilot]

Pull request overview

Copilot reviewed 8 out of 8 changed files in this pull request and generated no new comments.

Suppressed comments (2)

python/pyarrow/array.pxi:2297

• copy defaults to None, but the versioned export path passes it directly to ExportArrayVersionedToDLPack(..., c_bool copy). Cython cannot coerce None to c_bool, so arr.__dlpack__(max_version=(1, 0)) (or a consumer that passes copy=None) will raise a TypeError instead of exporting a capsule.

        # Currently no major version other than legacy 0 and current 1.3
        dlm_tensor = GetResultValue(ExportArrayVersionedToDLPack(self.sp_array, copy))
        return PyCapsule_New(dlm_tensor, 'dltensor_versioned', dlpack_versioned_pycapsule_deleter)

python/pyarrow/tensor.pxi:351

• copy defaults to None, but the versioned export path passes it directly to ExportTensorVersionedToDLPack(..., c_bool copy). Cython cannot coerce None to c_bool, so tensor.__dlpack__(max_version=(1, 0)) (or a consumer that passes copy=None) will raise a TypeError instead of exporting a capsule.

        # Currently no major version other than legacy 0 and current 1.3
        dlm_tensor = GetResultValue(ExportTensorVersionedToDLPack(self.sp_tensor, copy))
        return PyCapsule_New(dlm_tensor, 'dltensor_versioned', dlpack_versioned_pycapsule_deleter)

[Copilot]

Pull request overview

Copilot reviewed 9 out of 9 changed files in this pull request and generated no new comments.

Suppressed comments (2)

python/pyarrow/tensor.pxi:347

• copy is documented as forcing a data copy, but for the legacy (unversioned) capsule path (max_version is None or < (1, 0)), the argument is ignored and the export always shares data. This breaks the documented contract for copy=True. Either implement copying for legacy exports or explicitly reject copy when returning an unversioned capsule (e.g. require max_version >= (1, 0) for copy=True).

        if max_version is None or max_version < (1, 0):
            # Note: from March 2025 onwards, it's okay to raise BufferError here.
            # Still we keep the V0 version that was added in August 2026.
            legacy_tensor = GetResultValue(ExportTensorToDLPack(self.sp_tensor))
            return PyCapsule_New(legacy_tensor, 'dltensor', dlpack_pycapsule_deleter)

python/pyarrow/array.pxi:2293

• copy is documented as forcing a data copy, but when exporting the legacy (unversioned) capsule (max_version is None or < (1, 0)), the argument is ignored and the export always shares data. This violates the method’s documented contract and can surprise consumers attempting to avoid sharing/mutation issues. Either implement copying for the legacy path or reject copy when returning an unversioned capsule (e.g. require max_version >= (1, 0) for copy=True).

        if max_version is None or max_version < (1, 0):
            # Note: from March 2025 onwards, it's okay to raise BufferError here.
            # Still we keep the V0 version that was added in August 2026.
            legacy_tensor = GetResultValue(ExportArrayToDLPack(self.sp_array))
            return PyCapsule_New(legacy_tensor, 'dltensor', dlpack_pycapsule_deleter)

[AlenkaF]

@AlenkaF would you mind providing early feedback before I move on to the Python side.

Sorry for replying late, was off for a while.
Will have a look at the PR first thing tomorrow. Thanks for working on this!

[AlenkaF]

Happy to a pre-existing bug in ExportTensor being fixed! Also the new copy support enables the option to obtain a mutable buffer which might be desirable by some.

I added one nit and two comments regarding the copy keyword on the Python side. C++ looks perfect =)

[Copilot]

Pull request overview

Copilot reviewed 9 out of 9 changed files in this pull request and generated no new comments.

Suppressed comments (2)

cpp/src/arrow/c/dlpack.cc:122

• ExportBuffer always stamps DLManagedTensorVersioned.version with the compile-time DLPACK_MINOR_VERSION (currently 1.3). Since the Python __dlpack__(max_version=...) path doesn’t pass max_version into the C++ exporter, a consumer requesting e.g. max_version=(1, 0) will still receive a 1.3 capsule (and potentially 1.3-only flag bits). If consumers enforce max_version strictly, this can break interoperability. Consider plumbing the requested max_version (at least the minor) into the versioned export APIs and capping version/flags to the requested maximum, or else clarifying that max_version is only used to select legacy vs versioned capsules.

  if constexpr (std::is_same_v<DT, DLManagedTensorVersioned>) {
    ctx->tensor.version = {.major = DLPACK_MAJOR_VERSION, .minor = DLPACK_MINOR_VERSION};
    ctx->tensor.flags = p.flags;
  }

python/pyarrow/array.pxi:2263

• The dl_device docstring says tuple[enum.Enum, int], but __dlpack_device__ returns a plain (int, int) tuple (and tests assert (1, 0)). This mismatch can confuse callers that try to pass back an Enum instance.

        dl_device : tuple[enum.Enum, int], optional

[Copilot]

Pull request overview

Copilot reviewed 9 out of 9 changed files in this pull request and generated no new comments.

Suppressed comments (3)

python/pyarrow/array.pxi:2302

• In the versioned export path, copy == True will silently treat invalid copy values (e.g. copy='yes') as False, and it also accepts truthy non-bools (e.g. 1). The DLPack protocol expects copy to be None or a bool, so this should validate the argument and use an identity check (is True) to avoid surprising behavior. (Same applies to Tensor.__dlpack__.)

        # Currently no major version other than legacy 0 and current 1.3
        dlm_tensor = GetResultValue(
            ExportArrayVersionedToDLPack(self.sp_array, copy == True))
        return PyCapsule_New(dlm_tensor, 'dltensor_versioned', dlpack_versioned_pycapsule_deleter)

python/pyarrow/array.pxi:2301

• max_version is documented as the maximum DLPack version supported by the consumer, but the versioned branch always exports a 1.3 capsule (see the comment "current 1.3") even when max_version is lower (e.g. (1, 0)). Consumers that enforce the requested max minor version may reject the capsule. Consider plumbing the requested version through (or clamping the exported DLManagedTensorVersioned.version to min(supported, max_version)), in both Array and Tensor exports.

        if max_version is None or max_version < (1, 0):
            if copy is not None:
                raise BufferError(
                    f"The copy argument is not supported with legacy (pre 1.0) DLPack version."
                )
            # Note: from March 2025 onwards, it's okay to raise BufferError here.
            # Still we keep the V0 version that was added in August 2026.
            legacy_tensor = GetResultValue(ExportArrayToDLPack(self.sp_array))
            return PyCapsule_New(legacy_tensor, 'dltensor', dlpack_pycapsule_deleter)

        # Currently no major version other than legacy 0 and current 1.3
        dlm_tensor = GetResultValue(
            ExportArrayVersionedToDLPack(self.sp_array, copy == True))

cpp/src/arrow/c/dlpack.cc:118

• ExportBuffer unconditionally sets dl_tensor.shape / dl_tensor.strides from Vec::data(). For 0-d tensors (ndim == 0), the DLPack ABI allows (and some consumers expect) shape == NULL and strides == NULL; leaving them non-null can yield a pointer that is not safe to dereference despite ndim==0. Setting both to nullptr when p.ndim == 0 makes the export conform to the ABI comment in dlpack_abi.h and avoids edge-case UB.

  ctx->tensor.dl_tensor.ndim = p.ndim;
  ctx->tensor.dl_tensor.shape = ctx->shape.data();
  ctx->tensor.dl_tensor.byte_offset = 0;
  // Strides must be non-null when ndim > 0
  ctx->tensor.dl_tensor.strides = ctx->strides.data();