Skip to content

NVIDIA Triton Inference Server Organization

NVIDIA Triton Inference Server provides a cloud and edge inferencing solution optimized for both CPUs and GPUs.

This top level GitHub organization host repositories for officially supported backends, including TensorRT, TensorFlow, PyTorch, Python, ONNX Runtime, and OpenVino. The organization also hosts several popular Triton tools, including:

  • Model Analyzer: A tool to analyze the runtime performance of a model and provide an optimized model configuration for Triton Inference Server.

  • Model Navigator: a tool that provides the ability to automate the process of moving a model from source to optimal format and configuration for deployment on Triton Inference Server.

Getting Started

To learn about NVIDIA Triton Inference Server, refer to the Triton developer page and read our Quickstart Guide. Official Triton Docker containers are available from NVIDIA NGC.

Product Documentation

User documentation on Triton features, APIs, and architecture is located in the server documents on GitHub. A table of contents for the user documentation is located in the server README file.

Release Notes, Support Matrix, and Licenses information are available in the NVIDIA Triton Inference Server Documentation.

Examples

Specific end-to-end examples for popular models, such as ResNet, BERT, and DLRM are located in the NVIDIA Deep Learning Examples page on GitHub. Additional generic examples can be found in the server documents.

Feedback

Share feedback or ask questions about NVIDIA Triton Inference Server by filing a GitHub issue.

Pinned Loading

  1. server Public

    The Triton Inference Server provides an optimized cloud and edge inferencing solution.

    Python 8.8k 1.5k

  2. core Public

    The core library and APIs implementing the Triton Inference Server.

    C++ 115 104

  3. backend Public

    Common source, scripts and utilities for creating Triton backends.

    C++ 307 93

  4. client Public

    Triton Python, C++ and Java client libraries, and GRPC-generated client examples for go, java and scala.

    Python 600 235

  5. model_analyzer Public

    Triton Model Analyzer is a CLI tool to help with better understanding of the compute and memory requirements of the Triton Inference Server models.

    Python 454 78

  6. model_navigator Public

    Triton Model Navigator is an inference toolkit designed for optimizing and deploying Deep Learning models with a focus on NVIDIA GPUs.

    Python 196 26

Repositories

Showing 10 of 36 repositories
  • Rust 36 Apache-2.0 10 24 (8 issues need help) 31 Updated Feb 19, 2025
  • server Public

    The Triton Inference Server provides an optimized cloud and edge inferencing solution.

    Python 8,759 BSD-3-Clause 1,526 638 (3 issues need help) 66 Updated Feb 19, 2025
  • pytorch_backend Public

    The Triton backend for the PyTorch TorchScript models.

    C++ 143 BSD-3-Clause 45 0 5 Updated Feb 19, 2025
  • perf_analyzer Public
    C++ 43 BSD-3-Clause 11 6 12 Updated Feb 19, 2025
  • onnxruntime_backend Public

    The Triton backend for the ONNX Runtime.

    C++ 138 BSD-3-Clause 58 72 3 Updated Feb 18, 2025
  • openvino_backend Public

    OpenVINO backend for Triton.

    C++ 30 BSD-3-Clause 16 6 4 Updated Feb 18, 2025
  • triton_cli Public

    Triton CLI is an open source command line interface that enables users to create, deploy, and profile models served by the Triton Inference Server.

    Python 54 2 2 1 Updated Feb 18, 2025
  • vllm_backend Public
    Python 224 BSD-3-Clause 23 0 5 Updated Feb 18, 2025
  • tensorrtllm_backend Public

    The Triton TensorRT-LLM Backend

    Python 779 Apache-2.0 114 299 (1 issue needs help) 21 Updated Feb 18, 2025
  • python_backend Public

    Triton backend that enables pre-process, post-processing and other logic to be implemented in Python.

    C++ 588 BSD-3-Clause 158 0 12 Updated Feb 17, 2025