CUDA Onnx models

[deepinderdeol]

Hi - using CUDA onnx models with the onnx connector results in an exception.

Here's the message: CUDA execution provider is not enabled in this build.

1. Will these models be supported?

2. And will function calling work for onnx connector/models?

[nmoeller]

This error Message reads like you have an model optimized for GPU but using the CPU Package.
I think you need to install the correct package with the CUDA Execution provider to run the model.

Running Onnx with GPU is not so straight forward locally.

1. Make sure you have the correct CUDA & CUDNN Version installed , i am using 12.X and 9.X.

2. Then you have to install or build the corresponding ONNX Genai package for your execution provider
   If you are running python install pip install onnxruntime-genai-cuda, for dotNet it should be dotnet add package Microsoft.ML.OnnxRuntime.Gpu

I've tested this on my machine and this works for python.

[deepinderdeol]

Thanks @nmoeller.
I was just using the Semantic Kernel connector for onnx.

[deepinderdeol]

Looks like this issue/pull request should resolve the problem:
#9056
#9437

How about function calling? Is that something that could work with Onnx models?

[nmoeller]

Regarding Function Calling ,this is a bit more complex with SLM's but possible with semantic kernel. First you need a model that is capable of doing function calling for example Nexus Raven or you have to finetune your own model. Then you can overwrite the Classes in Semantic Kernel to adapt to the function calling behavior of your model.

Here is a python example overwriting the base class of a connector with Nexus Raven

https://github.com/microsoft/semantic-kernel/blob/main/python/samples/concepts/auto_function_calling/nexus_raven.py

If you want to transfer this to Onnx with Nexus Raven you would need to do the following things:

1. Convert the Nexus Raven to Onnx
2. Overwrite the OnnxBaseClass, to parse the string of the model into a FunctionCallContent Object.
3. Invoke the FunctionCallObject of SemanticKernel (Not sure how to do this in .Net in python we have kernel.invoke_function_call)

This is how i was able to do it in python.
But i am not sure if the python sample can be translated to .Net.

[RogerBarreto]

@nmoeller Currently the only version we have our C# connectors is the CPU enabled one. Attempting to inject a Cuda library using the CPU enabled connector will give an exception:

Unhandled exception. Microsoft.ML.OnnxRuntimeGenAI.OnnxRuntimeGenAIException: CUDA execution provider is not enabled in this build.

We are currently working on this issue.

• .Net: Bug: .NET Microsoft.SemanticKernel.Connectors.Onnx missing package release for CUDA and DirectML #9056

[RogerBarreto]

How about function calling? Is that something that could work with Onnx models?

@deepinderdeol GenAI library as of now doesn't support function calling.

We currently have an open discussion in their repository asking for this feature for integration.

• ONNX Function Calling Support onnxruntime-genai#969

[deepinderdeol]

Thanks @RogerBarreto