-
Notifications
You must be signed in to change notification settings - Fork 13
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Implement custom memory management for internal and output allocations #94
base: main
Are you sure you want to change the base?
Conversation
std::unique_ptr<nvinfer1::IGpuAllocator> gpuAllocator( | ||
new TensorRTLinearAllocator(kCURR_ALLOC_SIZE)); | ||
if (opts.useCustomGpuAllocator) { | ||
builder->setGpuAllocator(gpuAllocator.get()); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@christopherbate Here are the builder-side changes to allow the GPU allocator to be set up. We probably do not need this and will only be required for runtime allocations.
842ad29
to
dec20fe
Compare
private: | ||
RuntimeSessionOptions options; | ||
ExecutableView executable; | ||
|
||
std::unique_ptr<PinnedMemoryAllocator> pinnedMemoryAllocator; | ||
std::unique_ptr<AllocTracker> allocTracker; | ||
std::unique_ptr<ResourceTracker> resourceTracker; | ||
|
||
GpuAllocator* gpuAllocator; |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
It is unclear how to manage the lifetime. Should I use unique_ptr
instead of the raw pointer?
@@ -36,7 +36,7 @@ namespace mlirtrt::runtime { | |||
/// `main` function. It is assumed that `main` takes no arguments and returns an | |||
/// integer result (which is returned if the execution is successful). | |||
/// TODO: this should take a handle to a function for streaming output/errors. | |||
StatusOr<int64_t> runExecutorLuaScript(std::string_view luaScript); | |||
StatusOr<int64_t> runExecutorLuaScript(std::string_view luaScript, GpuAllocator* allocator); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Is it OK to add pass allocator
here?
//! Class to allocate memory for outputs with data-dependent shapes. The sizes | ||
//! of those are unknown so pre-allocation is not possible. | ||
//! | ||
class OutputAllocator : public nvinfer1::IOutputAllocator { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I might add this in a separate PR.
1ddb4b0
to
7b9a0f7
Compare
class GpuAllocator { | ||
public: | ||
GpuAllocator() = default; | ||
virtual ~GpuAllocator() = default; | ||
|
||
virtual StatusOr<void *> reallocate(void *baseAddr, uint64_t alignment, |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Temporarily remove it to simplify testing.
7b48086
to
0e094ec
Compare
ninja -C build/mlir-tensorrt check-mlir-executor
ninja -C build/mlir-tensorrt check-mlir-tensorrt-dialect
ninja -C build/mlir-tensorrt check-mlir-tensorrt
|
06102b6
to
94cbb5a
Compare
94cbb5a
to
fc7ca69
Compare
No description provided.