[Tracker] Onnx FE Support

[vivekkhandelwal1]

This issue is for the purpose of tracking all the ONNX Frontend Requirements.

Instructions for finding the models/setup:

• Linux server access
• Source location of tests
• IREE EP

Important Links

• e2eshark nightly reports
• CI test lists
• Onnx OP Support [tracking] ONNX Op Support #215
• Main Issues Tracker [Tracker] All the issue related with e2e shark test suite #812
• The torch OP support is tracked by [tracking] TorchToLinalg Op Support #347

ONNX model

Passing Summary

CPU

TOTAL TESTS = 2287

Stage	# Passing	% of Total	% of Attempted
Setup	2287	100.0%	100.0%
IREE Compilation	2128	93.0%	93.0%
Gold Inference	2128	93.0%	100.0%
IREE Inference Invocation	2036	89.0%	95.7%
Inference Comparison (PASS)	1587	69.4%	77.9%

GPU

TOTAL TESTS = 2287

Stage	# Passing	% of Total	% of Attempted
Setup	2284	99.9%	99.9%
IREE Compilation	1823	79.7%	79.8%
Gold Inference	1823	79.7%	100.0%
IREE Inference Invocation	1751	76.6%	96.1%
Inference Comparison (PASS)	1369	59.9%	78.2%

Fail Summary

CPU and GPU

TOTAL TESTS = 2338

Stage	CPU	GPU
Setup	0	3
IREE Compilation	159	461
Gold Inference	0	0
IREE Inference Invocation	92	72
Inference Comparison	449	382

Latest Status (Inference Pass/ Compile Pass/Total)

Item	Current (Oct 10)	Target (Oct 14 )
Pre-June-Shark test suite	22/29/32	100%
Vision int8 Models	52/78/78	100%
P0/P1 int8 CNN Models	417/486/486	100%
Hugging Face CNN FP32 Models	221/402/499	100%
Protected Models	14/23/25	100%
MIGraphX Models	17/22/29	100%
Hugging face non-CNN models	880/1146/1198	100%
IREE EP Models	19/19/35	100%
Onnx iree tests	764/1217(63%)	65%
Torch OP	871/1408 (62%)	65%
Total tests	3277/3840/5007

The Onnx lowering Lit Tests

View the op name from the tracker and then take out the lit test corresponding to that op in a seperate file, and run:

torch-mlir-opt --convert-torch-onnx-to-torch --torch-decompose-complex-ops --canonicalize --torch-backend-to-linalg-on-tensors-backend-pipeline test.mlir

Torch Op E2E Tests of torch-mlir

Take out the E2E test from the tracker and run:

python -m projects.pt1.e2e_testing.main -f <test_name> -v --config=onnx

ONNX Op Shark_TestSuite/iree_tests

Compile time Tests - #563
Runtime Tests - #583
To run the test, please follow:
build venv following here and run

iree-compile` iree_tests/onnx/node/generated/TEST_NAME/model.mlir -o test.vmfb --iree-hal-target-backends=llvm-cpu --mlir-print-ir-after-all

Models Shark_TestSuite/e2eshark

The E2EShark Model Tests are tracked through #566

First, follow setup instructions at https://github.com/nod-ai/SHARK-TestSuite/tree/main/e2eshark. No need to do the Turbine setup part as we are looking at onnx mode. Then, run this command (HF_TOKEN needed for llama, gemma model):

HF_TOKEN=your_token python run.py --torchmlirbuild /path/to/torch-mlir/build --ireebuild /path/to/iree-build --cachedir ~/.cache/huggingface --tests pytorch/models/<model_name> -r test-onnx --tolerance .001 .001 --mode onnx --report

[kumardeepakamd]

Owners, kindly add clear steps to reproduce failures and allow ability for contributors to take up a unique issue and work on fix to have more folks join in for this quality push. Great start! Let's do it.

[Shukla-Gaurav]

The IREE-EP related efforts are being tracked here: https://github.com/nod-ai/npu-benchmark/issues/2
Currently, we don't have any numbers related to model-passing rate. Once we have that, I will update that here as well. Thanks!