Onnx2tensorrt

Self hostable service to convert an ONNX model to TensorRT engine.

Deployment

Server deployment is mandatory for running this app. The fullstack app does not need to be deployed.

Server deployment

1. Create a Modal account. Low usage should fit within the $30 monthly free tier.

2. Create 2 Cloudflare R2 buckets with access key

3. Create a Modal secret:

ONNX_BUCKET=
TRT_BUCKET=
S3_ENDPOINT=
S3_KEY_ID=
S3_KEY=
REGION_NAME=auto

4. Create .env file in project root for Modal backend deployment:

APP_NAME=
MODAL_WORKSPACE=

5. Install python package and deploy:

poetry install
pnpm run deploy:modal

Attribution

• Modal - Serverless GPU FAAS platform.
• TensorRT - Build fast ML models for Nvidia GPU