[FLINK-38857][Model] Introduce a Triton inference module under flink-models #27385
Conversation
|
|
||
| # Triton | ||
|
|
||
| The Triton Model Function allows Flink SQL to call [NVIDIA Triton Inference Server](https://github.com/triton-inference-server/server) for real-time model inference tasks. |
There was a problem hiding this comment.
I have added some comments, can you ask on the dev list whether this requires a Flip please; to me it seems big enough to warrant a Flip.
github-actions
bot
added
the
community-reviewed
| ```sql | ||
| CREATE TEMPORARY VIEW movie_reviews(id, movie_name, user_review, actual_sentiment) | ||
| AS VALUES | ||
| (1, 'Great Movie', 'This movie was absolutely fantastic! Great acting and storyline.', 'positive'), |
There was a problem hiding this comment.
nit: I wonder whether -1, 0 and +1 would be more intuitive values.
|
|
||
| Here's an example `config.pbtxt` for a text classification model: | ||
|
|
||
| ```protobuf |
There was a problem hiding this comment.
I suggest we explicitly say that this should be in the text-classification/ folder.
| ├── text-classification/ | ||
| │ ├── config.pbtxt | ||
| │ └── 1/ | ||
| │ └── model.py # or model.onnx, model.plan, etc. |
There was a problem hiding this comment.
in the following example what file do we use for model.py
There was a problem hiding this comment.
Good question — this refers to the Triton Python backend model file.
In this example, model.py is the Python backend implementation located in the Triton model repository, specifically under:
text-classification/
├── config.pbtxt
└── 1/
└── model.py
The exact contents of model.py are not relevant to Flink itself. Flink interacts with the model only via the Triton HTTP/gRPC inference API, and does not load or execute the model code directly.
To avoid ambiguity, I will update the documentation to explicitly state that this file resides in the text-classification/ model directory.
featzhang
changed the title
3 participants
What is the purpose of the change
This PR introduces a new optional Triton inference module under
flink-models, enabling Flink to invoke external NVIDIA Triton Inference Server for batch-oriented model inference.The module implements a reusable runtime-level integration based on the existing model provider SPI, allowing users to define Triton-backed models via
CREATE MODELand execute inference throughML_PREDICTwithout modifying the Flink planner or SQL execution semantics.Brief change log
flink-model-tritonmodule underflink-modelsVerifying this change
Does this pull request potentially affect one of the following parts?
Documentation
docs/connectors/models/triton.mdRelated issues