From d9756f9f6d757bc3f0dc8302ff85598270ddd1bc Mon Sep 17 00:00:00 2001 From: Noah Kennedy Date: Wed, 14 Aug 2024 19:02:36 -0500 Subject: [PATCH] rfc: propose an FFI API This RFC covers the process aspects of this and largely defers discussion of the technical aspects, preferring to allow those to be incrementally addressed over time. --- tokio/docs/ffi.md | 106 ++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 106 insertions(+) create mode 100644 tokio/docs/ffi.md diff --git a/tokio/docs/ffi.md b/tokio/docs/ffi.md new file mode 100644 index 00000000000..598b76a82ba --- /dev/null +++ b/tokio/docs/ffi.md @@ -0,0 +1,106 @@ +# RFC: An FFI API for Tokio + +## Summary + +[summary]: #summary + +This RFC proposes the creation of an initially unstable FFI API for the Tokio crate. +In the interest of seeing implementation, usage, and feedback faster, the initial scope of this effort will be narrow +but will grow over time and evolve as we better explore the solution space. +This is expected to remain unstable for some time as it is a relatively new and unproven problem space, and the API is +expected to potentially evolve considerably over time. + +## Motivation + +[motivation]: #motivation + +While "Rewrite it in Rust" is a common trope, for many projects this is not realistic. +Many projects are too large to rewrite entirely or non-incrementally. +Thus, when adopted, Rust may often have to live alongside C or C++ in a codebase for a significant amount of +time, potentially permanently. + +Unfortunately, Tokio does not currently expose an FFI API which other languages can interact with. +This creates a significant barrier to the use of Tokio and the adoption of Rust within these systems. + +However, if we were to expose an FFI API, existing C/C++ event loops capable of doing so could be bridged to Tokio, +significantly easing this process. + +Long term, it may even be desirable to allow the reverse - Tokio could act as an interface into other runtimes, +however that is much dicier and will be left out of the scope of this proposal. + +## Implementation Design + +[guide-level-explanation]: #implementation-design + +Within the Tokio crate, an `ffi-experimental` subdirectory will be added containing three subdirectories, +`public-include`, containing header files specifying the public interface of the crate, `examples`, containing examples +of usage which should be tested for compilation in CI, and `tests`, containing source code for a library consisting of +test cases which we can build, link to, and invoke when calling Tokio's integration tests behind a special `--cfg` +flag. +A Makefile with rules for invoking tests and running examples should also be added. +Within the Tokio crate, an `ffi_experimental` module should be added containing extern bindings for exposed APIs. + +It is assumed initially that users will be vendoring this code and managing builds themselves, so any build system +logic we have should be focused on our own testing and usage. + +Header files and extern "C" bindings should be written and designed manually and not autogenerated using tooling. +Header files will be targeted at [C17](https://en.cppreference.com/w/c/17). + +## Process and Lifecycle + +[reference-level-explanation]: #process-and-lifecycle + +This API should be implemented in several phases. +The API should be evolved and design changes and decisions should be made as we go along. +The API should be kept clearly marked as unstable, although we should also avoid and very clearly document breaking +changes where we make them, and ideally phase these changes by preferring new functions over changing existing ones. + +The initial phase should be the implementation of core features: runtime construction and operation, as well as task +spawning, waking, and execution, timers, signals, and IO via AsyncFd. +In this phase, the goal is to ship the minimum viable product capable of being useful and gather feedback on it to feed +into subsequent phases and amend plans as needed. + +After the core components have been shipped, we can begin exploring additional APIs which may be useful to expose, and +can continue iterating on existing components. +New APIs may include exposing additional modules within Tokio, or even useful bridge-layer abstractions, like coroutine +support. + +This RFC avoids discussions on the precise API designs themselves in order to avoid and defer bike-shedding and to +establish early consensus about the usefulness of this initiative. + +## Drawbacks + +[drawbacks]: #drawbacks + +This may be a significant undertaking, and it may take time and iteration to get this right. + +## Rationale and alternatives + +[rationale-and-alternatives]: #rationale-and-alternatives + +We could insist that users maintain their own bridges, but that would potentially limit the scope of what can be +bridged, impose a significant burden on users wishing to do so, and lead to wasted, duplicative effort. + +## Prior art + +[prior-art]: #prior-art + +There is relatively limited prior art here - we are treading on a significant amount of new ground. +As a result, this RFC heavily emphasizes the evolvability of any design. + +## Unresolved questions + +[unresolved-questions]: #unresolved-questions + +All details of the specific APIs themselves are left unresolved here, and should be handled individually by API. + +Also left unresolved through this RFC are questions of modifications to Tokio itself needed may be needed. +For example, do we need to expose cooperatively abortable tasks as a construct, or support dynamic linking? + +## Future possibilities + +[future-possibilities]: #future-possibilities + +As this initiative evolves and progresses, we can add feedback for more APIs within Tokio, add support for C++ +coroutines, and potentially even support allowing Tokio to act as an interface to other runtimes, allowing Tokio to be +used as a runtime for runtimes like Folly, rather than just a backend.