Hello world tutorial error: RuntimeException: [ONNXRuntimeError] : 6 : RUNTIME_EXCEPTION : Exception during initialization: CHECK failed: it != end(): key not found: profile

[ChristiaanBoe]

Hello everyone,

I am having some trouble with my Minisforum um790 pro.

For context, I have followed the steps on both the automated installation (https://ryzenai.docs.amd.com/en/latest/inst.html) using the install.bat script as well as the manual one (https://ryzenai.docs.amd.com/en/latest/manual_installation.html). The only exception regarding following the steps was that I used Visual Studio 2022 instead of 2019 as it is hidden behind a Microsoft subscription.

I did this by changing the following line from the bat script
reg query "HKEY_LOCAL_MACHINE\SOFTWARE\WOW6432Node\Microsoft\VisualStudio\16.0" >nul 2>&1
into
reg query "HKEY_LOCAL_MACHINE\SOFTWARE\WOW6432Node\Microsoft\VisualStudio\17.0" >nul 2>&1.

I run the automated installation using vs code, in the same manner as the youtube tutorial resulting in the following error message:

RuntimeException                          Traceback (most recent call last)
Cell In[8], line 9
      6 aie_options = onnxruntime.SessionOptions()
      7 aie_options.enable_profiling = True
----> 9 aie_session = onnxruntime.InferenceSession(
     10     onnx_model_path,
     11     providers = ['VitisAIExecutionProvider'],
     12     sess_options=aie_options,
     13     provider_options=[{'config_file': config_file_path}]
     14 )
     16 start = timer()
     17 ryzen_outputs = aie_session.run(None, {'input': input_data})

File c:\Users\chris\anaconda3\envs\ryzenai-1.1-20240530-182215\lib\site-packages\onnxruntime\capi\onnxruntime_inference_collection.py:383, in InferenceSession.__init__(self, path_or_bytes, sess_options, providers, provider_options, **kwargs)
    380 disabled_optimizers = kwargs["disabled_optimizers"] if "disabled_optimizers" in kwargs else None
    382 try:
--> 383     self._create_inference_session(providers, provider_options, disabled_optimizers)
    384 except (ValueError, RuntimeError) as e:
    385     if self._enable_fallback:

File c:\Users\chris\anaconda3\envs\ryzenai-1.1-20240530-182215\lib\site-packages\onnxruntime\capi\onnxruntime_inference_collection.py:435, in InferenceSession._create_inference_session(self, providers, provider_options, disabled_optimizers)
    432     disabled_optimizers = set(disabled_optimizers)
    434 # initialize the C++ InferenceSession
--> 435 sess.initialize_session(providers, provider_options, disabled_optimizers)
    437 self._sess = sess
    438 self._sess_options = self._sess.session_options

RuntimeException: [ONNXRuntimeError] : 6 : RUNTIME_EXCEPTION : Exception during initialization: CHECK failed: it != end(): key not found: profile

In turn the manual installation is run via an jupyter environment resulting in the following error message:

RuntimeException                          Traceback (most recent call last)
Cell In[6], line 9
      6 aie_options = onnxruntime.SessionOptions()
      7 aie_options.enable_profiling = True
----> 9 aie_session = onnxruntime.InferenceSession(
     10     onnx_model_path,
     11     providers = ['VitisAIExecutionProvider'],
     12     sess_options=aie_options,
     13     provider_options=[{'config_file': config_file_path}]
     14 )
     16 start = timer()
     17 ryzen_outputs = aie_session.run(None, {'input': input_data})

File ~\anaconda3\envs\Ryzentest\lib\site-packages\onnxruntime\capi\onnxruntime_inference_collection.py:383, in InferenceSession.__init__(self, path_or_bytes, sess_options, providers, provider_options, **kwargs)
    380 disabled_optimizers = kwargs["disabled_optimizers"] if "disabled_optimizers" in kwargs else None
    382 try:
--> 383     self._create_inference_session(providers, provider_options, disabled_optimizers)
    384 except (ValueError, RuntimeError) as e:
    385     if self._enable_fallback:

File ~\anaconda3\envs\Ryzentest\lib\site-packages\onnxruntime\capi\onnxruntime_inference_collection.py:435, in InferenceSession._create_inference_session(self, providers, provider_options, disabled_optimizers)
    432     disabled_optimizers = set(disabled_optimizers)
    434 # initialize the C++ InferenceSession
--> 435 sess.initialize_session(providers, provider_options, disabled_optimizers)
    437 self._sess = sess
    438 self._sess_options = self._sess.session_options

RuntimeException: [ONNXRuntimeError] : 6 : RUNTIME_EXCEPTION : Exception during initialization: CHECK failed: it != end(): key not found: profile

I have also attached the error log of the jupyter notebook in ryzenlog.txt.
ryzenlog.txt

If someone could help me find out what I did wrong it would very much be appreciated.

Best regards

[farshadghodsian]

Someone had the same error in a previous issue and was able to resolve it. Worth checking if this also helps resolve your issue: #59 (comment)

[ChristiaanBoe]

I gave this a try and it resulted in my kernel completely crashing unfortunately

[vgodsoe]

Hi @ChristiaanBoe,
I was able to reproduce your issue, and this is what fixed it for me. If you'd like to give it a try and let me know if it works on your end, that would be great!

In the quantization section in Step 3, update it to the following. As well, you'll want to restart the Jupyter Notebook before rerunning all the cells so that there's no cached data.

vai_q_onnx.quantize_static(
    input_model_path,
    output_model_path,
    calibration_data_reader=None,
    quant_format=vai_q_onnx.QuantFormat.QDQ,
    calibrate_method=vai_q_onnx.PowerOfTwoMethod.MinMSE,
    activation_type=vai_q_onnx.QuantType.QUInt8,
    weight_type=vai_q_onnx.QuantType.QInt8,
    enable_ipu_cnn=True,
    extra_options={'ActivationSymmetric': True}
)

Thanks,
Victoria

[ChristiaanBoe]

Hello @vgodsoe ,

Thank you so much for your help so far!

I have tried using your method resulting in the following onnx files:
ONNX files.zip

Unfortunatly using both the automated and the manual install I am still greeted with an onnxRuntime error however it appears I now get different outputs.

I also used the #59 method for both automated and manual methods.
Manual error normal:

Cell In[6], line 9
      6 aie_options = onnxruntime.SessionOptions()
      7 aie_options.enable_profiling = True
----> 9 aie_session = onnxruntime.InferenceSession(
     10     onnx_model_path,
     11     providers = ['VitisAIExecutionProvider'],
     12     sess_options=aie_options,
     13     provider_options=[{'config_file': config_file_path}]
     14 )
     16 start = timer()
     17 ryzen_outputs = aie_session.run(None, {'input': input_data})

File ~\anaconda3\envs\Ryzentest\lib\site-packages\onnxruntime\capi\onnxruntime_inference_collection.py:383, in InferenceSession.__init__(self, path_or_bytes, sess_options, providers, provider_options, **kwargs)
    380 disabled_optimizers = kwargs["disabled_optimizers"] if "disabled_optimizers" in kwargs else None
    382 try:
--> 383     self._create_inference_session(providers, provider_options, disabled_optimizers)
    384 except (ValueError, RuntimeError) as e:
    385     if self._enable_fallback:

File ~\anaconda3\envs\Ryzentest\lib\site-packages\onnxruntime\capi\onnxruntime_inference_collection.py:435, in InferenceSession._create_inference_session(self, providers, provider_options, disabled_optimizers)
    432     disabled_optimizers = set(disabled_optimizers)
    434 # initialize the C++ InferenceSession
--> 435 sess.initialize_session(providers, provider_options, disabled_optimizers)
    437 self._sess = sess
    438 self._sess_options = self._sess.session_options

RuntimeException: [ONNXRuntimeError] : 6 : RUNTIME_EXCEPTION : Exception during initialization: CHECK failed: it != end(): key not found: profile

Manual error using #59

RuntimeException                          Traceback (most recent call last)
Cell In[10], line 23
     15 cache_dir = Path(os.path.join(os.getcwd(), 'cache'))
     17 provider_options = [{
     18 'config_file': os.path.join('..', path, 'vaip_config.json'), # Adjust the config file path as necessary
     19 'cacheDir': str(cache_dir), # Convert the cache directory Path object to string
     20 'cacheKey': 'modelcachekey_quick'
     21 }]
---> 23 aie_session = ort.InferenceSession(model, providers=providers,
     24 provider_options=provider_options)
     26 start = timer()
     27 ryzen_outputs = aie_session.run(None, {'input': input_data})

File ~\anaconda3\envs\ryzentestweird\lib\site-packages\onnxruntime\capi\onnxruntime_inference_collection.py:383, in InferenceSession.__init__(self, path_or_bytes, sess_options, providers, provider_options, **kwargs)
    380 disabled_optimizers = kwargs["disabled_optimizers"] if "disabled_optimizers" in kwargs else None
    382 try:
--> 383     self._create_inference_session(providers, provider_options, disabled_optimizers)
    384 except (ValueError, RuntimeError) as e:
    385     if self._enable_fallback:

File ~\anaconda3\envs\ryzentestweird\lib\site-packages\onnxruntime\capi\onnxruntime_inference_collection.py:435, in InferenceSession._create_inference_session(self, providers, provider_options, disabled_optimizers)
    432     disabled_optimizers = set(disabled_optimizers)
    434 # initialize the C++ InferenceSession
--> 435 sess.initialize_session(providers, provider_options, disabled_optimizers)
    437 self._sess = sess
    438 self._sess_options = self._sess.session_options

RuntimeException: [ONNXRuntimeError] : 6 : RUNTIME_EXCEPTION : Exception during initialization: attrs_create_cif
attrs_create_cif
(unknown)
__NLG_Return2
RtlCaptureContext2
initialize_onnxruntime_vitisai_ep
compile_onnx_model_vitisai_ep_with_options
PyInit_onnxruntime_pybind11_state
PyInit_onnxruntime_pybind11_state
PyInit_onnxruntime_pybind11_state
PyInit_onnxruntime_pybind11_state
PyInit_onnxruntime_pybind11_state
PyInit_onnxruntime_pybind11_state
PyInit_onnxruntime_pybind11_state
PyInit_onnxruntime_pybind11_state
PyInit_onnxruntime_pybind11_state
PyInit_onnxruntime_pybind11_state
PyInit_onnxruntime_pybind11_state
PyInit_onnxruntime_pybind11_state
public: void __cdecl pybind11::error_already_set::discard_as_unraisable(class pybind11::object) __ptr64
_PyArg_ParseTuple_SizeT
_PyObject_MakeTpCall
_PyErr_FormatFromCauseTstate
_Py_NewReference
_PyEval_EvalFrameDefault
_PyFunction_Vectorcall
_Py_NewReference
_PyEval_EvalFrameDefault
_PyFunction_Vectorcall
_PyFunction_Vectorcall
PyCode_Optimize
_Py_NewReference

Resulting in the following error log:
errorlog manual #59.txt

Using the automated method normally I got the following error messages:

RuntimeException                          Traceback (most recent call last)
Cell In[9], line 9
      6 aie_options = onnxruntime.SessionOptions()
      7 aie_options.enable_profiling = True
----> 9 aie_session = onnxruntime.InferenceSession(
     10     onnx_model_path,
     11     providers = ['VitisAIExecutionProvider'],
     12     sess_options=aie_options,
     13     provider_options=[{'config_file': config_file_path}]
     14 )
     16 start = timer()
     17 ryzen_outputs = aie_session.run(None, {'input': input_data})

File ~\anaconda3\envs\ryzentestweird\lib\site-packages\onnxruntime\capi\onnxruntime_inference_collection.py:383, in InferenceSession.init(self, path_or_bytes, sess_options, providers, provider_options, **kwargs)
    380 disabled_optimizers = kwargs["disabled_optimizers"] if "disabled_optimizers" in kwargs else None
    382 try:
--> 383     self._create_inference_session(providers, provider_options, disabled_optimizers)
    384 except (ValueError, RuntimeError) as e:
    385     if self._enable_fallback:

File ~\anaconda3\envs\ryzentestweird\lib\site-packages\onnxruntime\capi\onnxruntime_inference_collection.py:435, in InferenceSession._create_inference_session(self, providers, provider_options, disabled_optimizers)
    432     disabled_optimizers = set(disabled_optimizers)
    434 # initialize the C++ InferenceSession
--> 435 sess.initialize_session(providers, provider_options, disabled_optimizers)
    437 self._sess = sess
    438 self._sess_options = self._sess.session_options

RuntimeException: [ONNXRuntimeError] : 6 : RUNTIME_EXCEPTION : Exception during initialization: CHECK failed: it != end(): key not found: profile

With the following log:
errorlog automated.txt

Using the automated method using #59 I simply get a kernel shutdown

With the following log:
errorlog automated #59.txt

I feel like the problem might be with finding the the voe-4.0-win_amd64 folder.
Victoria thank you again for all your support!

Christiaan

[ryomo]

I had the same issue, which is fixed by this commit b361b5b

[ChristiaanBoe]

Hey @vgodsoe @ryomo after some fiddling and reinstalling Windows 11 I got the NPU to work (I think). I still find some weird bugs, as I first have to run the NPU execution cell, quantize the onnx file again and then execute that file via the NPU to get it to work. I was also a bit suprised regarding the low speed of the NPU on the minisforum. It usually is about two times as slow and at best of times the inference times are equal (see the attached image), is this the same result you guys get from the hello world notebook running on the Minisforum um790 pro?

Edit: the NPU is faster then CPU once the size of the fully connected layer increases

[ryomo]

Hi.
I get the same results from this hello world model.
I guess that the initial aie_session.run() includes some kind of build time, which is longer than cpu_session.run().
So, the second run gets faster and nearly same results between the NPU and CPU in this simple model.

FYI, other quantized models which I'm working on are showing faster inference speeds via the NPU than via the CPU. (But super low accuracy for now 😢 )

[ChristiaanBoe]

Hey @ryomo @vgodsoe

Your work seems super interesting, to me the runtime seems to have quite a bit of variance, I am looking forward to seeing your solution!

What I noticed so far is that the IPU really shines compares to GPUs when it comes to large models with larges input sizes as it can use the more numerous CPU ram as opposed to CUDA RAM.

As I used the manual installation method one thing I noticed is that in my global variables NUM_OF_DPU_RUNNERS=1 could this be higher for the Minisforum um790pro, I am a bit afraid of changing anything at this point lest my programming environment shatters like a precious crystal statue.

[ryomo]

Now I have converted hello_world.ipynb to a normal Python script to check what's going on.
It prints more logs and shows [Vitis AI EP] No. of Operators : CPU 7, which seems to indicate that it's not working on the NPU lol.

I am a bit afraid of changing anything at this point lest my programming environment shatters like a precious crystal structure.

I completely agree.

[vgodsoe]

Hi @ryomo and @ChristiaanBoe

Thank you for your feedback! Love to see them. I checked in a new hello world example, and I wanted to share a couple of updates that will hopefully help!

1. If the there's only one convolutional layer, the compiler for the NPU automatically decides to only run the model on the NPU. I've updated the hello_world example with 2 conv layers and now the compiler picks it up for the NPU and you should see ops running on the NPU.
2. With the updated Jupyter Notebook, I've also added a Python script to make it easier for developers running the example to do debug or update the code. Sometimes, Jupyter Notebook doesn't show the full output messages, so this script is useful if you want to see all of the console updates.
3. In both the Jupyter Notebook and the Python script, I've specified a "cache" folder. This will store the compiled model and the compilation report. These are great for being able to see what ops are running on the CPU vs. the NPU. The updated example will generate a report for the NPU.
4. As @ryomo noted earlier, on the first execution on the NPU, the compiler is called for the model so the initial run will be slower. Subsequent runs should show the model more performant (since it used the compiled model from the cache), however, being a small "hello_world" model, you won't really see the performance gains in this example.
5. If you did want to play with the NUM_OF_DPU_RUNNERS, 2 and 4 are good options. Here's how you can add it into the code os.environ['NUM_OF_DPU_RUNNERS'] = "4"

Thanks,
Victoria

[ryomo]

Thank you for your clarifications and updates! The detailed explanations are very helpful.
And I see ops running on the NPU.

[ChristiaanBoe]

Hello @vgodsoe ,

Thank you so much for the clarifications and the continued support :)

I have been playing around with the NUM_OF_DPU_RUNNERS: rather surprisingly I found that both the NPU as well as the CPU appear to speed up roughly the same amount, is this an expected result?

All the best,

Christiaan