Onnx - Consider as way to share models

[cjerzak]

https://en.wikipedia.org/wiki/Open_Neural_Network_Exchange

https://huggingface.co/blog/convert-transformers-to-onnx

[beniaminogreen]

Hi Connor,

Did you manage to look into this? I am thinking of experimenting with this rust wrapper for the ONNX runtime as a way to provide fast inference in R for ML models without needing a python install. If we are able to export the models you have built using the ONNX format, we might be able to quickly perform inference using this approach.

Best,
Ben

[cjerzak]

true. i haven't taken a close look but will do so later this/next week! we do want to simplify the inference pipeline...

[beniaminogreen]

Just got back from travels and after a long time working out the compilation + some dependency steps, I am excited to report that I am able to deploy GPT-2 on my laptop using this approach.

Seems like this could be a very promising avenue for deploying our models if they are easy to translate into the appropriate format. Ideally, we could pre-process everything in R, then pass the data to the ONNX runtime to do the difficult steps efficiently. Will provide updates as I make more progress.

[cjerzak]

very cool! yes, minimizing dependencies for people will be very important... happy to talk anytime this week.

…

On Mon, Jul 22, 2024 at 10:43 PM Ben Green ***@***.***> wrote: Just got back from travels and after a long time working out the compilation + some dependency steps, I am excited to report that I am able to deploy GPT-2 on my laptop using this approach. Seems like this could be a very promising avenue for deploying our models if they are easy to translate into the appropriate format. Ideally, we could pre-process everything in R, then pass the data to the ONNX runtime to do the difficult steps efficiently. Will provide updates as I make more progress. — Reply to this email directly, view it on GitHub <#2 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AEZEOAENUVUBNXXOGAI5N4TZNW7LNAVCNFSM6AAAAABENOKIU6VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDENBUGE2DGNRQGY> . You are receiving this because you authored the thread.Message ID: ***@***.***>