Repository navigation
[cuda.core] Add support for host_launch (host callback nodes / host function launches) #2058
Description
Feature Request
Add a host_launch (or equivalent) API to cuda.core that allows scheduling
Python callables (or C function pointers) to execute on the host as part of a
stream's work order. This is the cuLaunchHostFunc / cudaLaunchHostFunc
path (and its graph-node counterpart, host nodes via cuGraphAddHostNode).
Motivation
cuda.core currently exposes launch(...) for device kernels but has no
symmetric primitive for host work. This makes it impossible to express
mixed host/device work ordering in pure cuda.core terms — users must drop
to cuda.bindings for cuLaunchHostFunc, which breaks the cuda.core
abstraction boundary (streams, events, graphs).
Common use cases:
- Logging / progress callbacks ordered against GPU work without host-side
stream synchronization. - Triggering Python-side state transitions (e.g. buffer release, metric
updates) at a specific point in a stream. - Host nodes in CUDA graphs for workflows that need host-side compute or
notification steps between kernels.
Proposed Scope
- A top-level
host_launch(stream, fn, *args, **kwargs)(or
stream.launch_host(fn, ...)) that wrapscuLaunchHostFunc. - A corresponding graph node type (
HostNode) added to
cuda.core.graph._subclasses, alongside the existingEmptyNode,
MemcpyNode, etc. - Clear documentation of the callback threading / reentrancy restrictions
imposed by the CUDA driver (host functions run on an internal driver
thread; must not call any CUDA API). - An example under
cuda_core/examples/demonstrating a host callback
ordered between two kernels. - API reference entries in
cuda_core/docs/source/api.rst.
Related
- Driver API:
cuLaunchHostFunc,cuGraphAddHostNode - Runtime API:
cudaLaunchHostFunc - Part of cuda.core feature audit gap list (Nov 2025).
Activity
kkraus14 commentedon May 14, 2026
One implementation concern: Python host callbacks can deadlock through CUDA-driver/GIL lock-order inversion if a CUDA driver callback thread tries to acquire the GIL while another Python thread holds the GIL and is blocked in a CUDA call. We should clearly document that host callbacks must not call CUDA APIs directly or indirectly, and that CUDA-using Python libraries must release the GIL around CUDA calls.
leofang commentedon Jul 9, 2026
@Andy-Jost, other than a
host_launchfree function this task is basically done, right? Given the deadlock concern, how useful is it for passing a Python function tohost_launch?Andy-Jost commentedon Jul 9, 2026
The graph infrastructure is in place and host callbacks can be added to graphs though explicit construction. This issue is correct in pointing out there's no freestanding
host_launchrepresenting host callbacks. That would be useful in its own right and also as a way to introduce host callbacks into graphs via stream capture.The deadlock concern is real, but it's something to be documented rather than a blocker. CUDA docs clearly state that host callbacks may not call into CUDA. Separate from that, I'd like to confirm whether the driver ever invokes a host callback while holding any driver lock.
0z5a commentedon Sep 20, 2026
Cross-posting a question from #311, since this issue owns the overlapping design and is assigned here.
Is any independent scope intended for #311 beyond this issue? Concretely: ordinary (non-graph) stream callback ordering and lifetime, owning vs borrowed stream handles, multi-device/multi-stream isolation, and exception reporting for that path. If your
host_launchwork already covers those, I will leave #311 alone; if part of it is out of scope here, I would take exactly that slice (tests and contract only, no second public API).Andy-Jost commentedon Oct 5, 2026
@0z5a There is no independent scope in #311. Everything on your list is the stream path of
host_launchin this issue: the function enqueues the callback on a stream,cuda.coreowns the callable and its user data until the driver has run them, errors follow the existing host-callback trampoline (a Python exception cannot cross into the driver and goes to the unraisable-exception hook), and the multi-stream and multi-device cases are ordinary tests of that function. The stream-side tests in your matrix need the entry point, so they are written with it. Please do not start an implementation; this issue is in progress.#2925 covers behavior that exists on
maintoday and is reviewed on its own.srimanachanta commentedon Oct 27, 2026
A use case for the stream path of
host_launchthat a Python callable does not cover: a host copy that runs in stream order without the GIL.Our library, a CUDA C++ solver with nanobind bindings, copies large device results into ordinary NumPy arrays through a pinned staging ring (_transfer.py). Each chunk is a device-to-pinned
memcpyAsyncfollowed by a host callback that copies the pinned slot into the NumPy array. The callback runs on the driver's callback thread while the main thread keeps queuing GPU work. Pinned staging measured 56 GB/s, against 20 GB/s for a direct device-to-pageable copy.The callback must not take the GIL. A Python callable goes through
_py_host_trampoline(with gil), so every chunk would contend with the thread that queues work, and it runs into the lock-order risk raised above. So we keep a nativehost_copy_asyncin our extension only for this (bindings.cpp): acudaLaunchHostFuncwhose callback is amemcpy.host_launchwith a ctypesCFUNCTYPE(None, c_void_p)anduser_dataalready avoids the GIL, since_resolve_host_callbackpasses such a pointer to the driver as is. What is missing is a function to point it at. A pure-Python package has nomemcpywith theCUhostFnsignature, so it would still need a compiled extension of its own.Two options that would remove that, either fine for us:
stream.copy_host(dst, src, nbytes)or ahost_launchhelper backed by a C callback insidecuda.core. It never touches Python, so it needs no GIL.Buffer.copy_to(numpy_array, stream=..., staging=PinnedMemoryResource), that does the pinned ring internally.Either would let us drop
host_copy_asyncfrom our extension. I'm happy to implement one of them on top ofhost_launchonce it lands, if you want it.