Model test failed #421
Comments
|
this reminds me that we should probably add the conda env the test was run in (output of
yeah, for most pytorch and onnx models the dev env works (any env with an up-to-date pytorch/onnxruntime package gets you very far for their great backwardcompatibility...) there is another extra |
|
jtlyk, affable-shark also does not pass in the current dev env, as the model's onnx weights cannot handle a batch size larger than 1 ❌ Run onnx inference for parametrized inputs
❌ weights.onnx [ONNXRuntimeError] : 2 : INVALID_ARGUMENT : Got
invalid dimensions for input: input.1 for the
following indices
index: 2 Got: 64 Expected: 256
index: 3 Got: 64 Expected: 256
Please fix either the inputs/outputs or the model.
older core versions just didn't test with a batch size >1, so this wasn't caught before. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Getting the following issue.
Noticed that it's because of the environment is not configured correctly, is there a universal conda env which can handle not all but most of the models? Should I just use the dev env?
cc @FynnBe
See error log
``` testing affable-shark... 2024-09-12 17:15:13.365 | INFO | bioimageio.spec._internal.io_utils:open_bioimageio_yaml:112 - loading affable-shark from https://uk1s3.embassy.ebi.ac.uk/public-datasets/bioimage.io/affable-shark/1.1/files/rdf.yaml Downloading data from 'https://uk1s3.embassy.ebi.ac.uk/public-datasets/bioimage.io/affable-shark/1.1/files/rdf.yaml' to file '/home/jovyan/.cache/bioimageio/c1d5091838f82f59738702f2d8a938ed-rdf.yaml'. 100%|█████████████████████████████████████| 4.22k/4.22k [00:00<00:00, 8.48MB/s] Downloading data from 'https://uk1s3.embassy.ebi.ac.uk/public-datasets/bioimage.io/affable-shark/1.1/files/zero_mean_unit_variance.ijm' to file '/home/jovyan/.cache/bioimageio/3c123c5162a87c48333c80f7389e99a7-zero_mean_unit_variance.ijm'. 100%|█████████████████████████████████████████| 845/845 [00:00<00:00, 1.29MB/s] computing SHA256 of 3c123c5162a87c48333c80f7389e99a7-zero_mean_unit_variance.ijm Downloading data from 'https://uk1s3.embassy.ebi.ac.uk/public-datasets/bioimage.io/affable-shark/1.1/files/documentation.md' to file '/home/jovyan/.cache/bioimageio/8c032152668b4aaa68f99fb56c067631-documentation.md'. 100%|█████████████████████████████████████| 1.57k/1.57k [00:00<00:00, 3.45MB/s] SHA256 hash of downloaded file: 9b8330e6fae8403efc7a44bb7365186c14fb6222b608791baab7a4e0352dbdcd Use this value as the 'known_hash' argument of 'pooch.retrieve' to ensure that the file hasn't changed if it is downloaded again in the future. Downloading data from 'https://uk1s3.embassy.ebi.ac.uk/public-datasets/bioimage.io/affable-shark/1.1/files/test_input_0.npy' to file '/home/jovyan/.cache/bioimageio/93e503596d7d0bbf5b3937cf1a80402b-test_input_0.npy'. 100%|████████████████████████████████████████| 262k/262k [00:00<00:00, 830MB/s] computing SHA256 of 93e503596d7d0bbf5b3937cf1a80402b-test_input_0.npy (result: c Downloading data from 'https://uk1s3.embassy.ebi.ac.uk/public-datasets/bioimage.io/affable-shark/1.1/files/sample_input_0.tif' to file '/home/jovyan/.cache/bioimageio/baeb9b3639733aa6ad5f8305149a3585-sample_input_0.tif'. 100%|██████████████████████████████████████| 65.8k/65.8k [00:00<00:00, 260MB/s] computing SHA256 of baeb9b3639733aa6ad5f8305149a3585-sample_input_0.tif (result: Downloading data from 'https://uk1s3.embassy.ebi.ac.uk/public-datasets/bioimage.io/affable-shark/1.1/files/test_output_0.npy' to file '/home/jovyan/.cache/bioimageio/858a63339bdb8f80ff310e077a5079bb-test_output_0.npy'. 100%|███████████████████████████████████████| 524k/524k [00:00<00:00, 2.06GB/s] computing SHA256 of 858a63339bdb8f80ff310e077a5079bb-test_output_0.npy (result: Downloading data from 'https://uk1s3.embassy.ebi.ac.uk/public-datasets/bioimage.io/affable-shark/1.1/files/sample_output_0.tif' to file '/home/jovyan/.cache/bioimageio/d22d18e9c3592839985cac2534ac4617-sample_output_0.tif'. 100%|███████████████████████████████████████| 525k/525k [00:00<00:00, 1.51GB/s] computing SHA256 of d22d18e9c3592839985cac2534ac4617-sample_output_0.tif (result Downloading data from 'https://uk1s3.embassy.ebi.ac.uk/public-datasets/bioimage.io/affable-shark/1.1/files/weights.onnx' to file '/home/jovyan/.cache/bioimageio/4f6534376602b6e094e1895b93c4ee74-weights.onnx'. 100%|████████████████████████████████████████| 116M/116M [00:00<00:00, 370GB/s] computing SHA256 of 4f6534376602b6e094e1895b93c4ee74-weights.onnx (result: df913 Downloading data from 'https://uk1s3.embassy.ebi.ac.uk/public-datasets/bioimage.io/affable-shark/1.1/files/unet.py' to file '/home/jovyan/.cache/bioimageio/760a77af34fd5e3e1eebeff7e0669585-unet.py'. 100%|█████████████████████████████████████| 20.9k/20.9k [00:00<00:00, 41.7MB/s] computing SHA256 of 760a77af34fd5e3e1eebeff7e0669585-unet.py (result: 7f5b15948e Downloading data from 'https://uk1s3.embassy.ebi.ac.uk/public-datasets/bioimage.io/affable-shark/1.1/files/weights.pt' to file '/home/jovyan/.cache/bioimageio/78e469078bf5116258ceaf6f41001dd7-weights.pt'. 100%|████████████████████████████████████████| 116M/116M [00:00<00:00, 128GB/s] computing SHA256 of 78e469078bf5116258ceaf6f41001dd7-weights.pt (result: 608f52c Downloading data from 'https://uk1s3.embassy.ebi.ac.uk/public-datasets/bioimage.io/affable-shark/1.1/files/weights-torchscript.pt' to file '/home/jovyan/.cache/bioimageio/d44de80414e1fd1414b97beaaed935f5-weights-torchscript.pt'. 100%|████████████████████████████████████████| 116M/116M [00:00<00:00, 252GB/s] computing SHA256 of d44de80414e1fd1414b97beaaed935f5-weights-torchscript.pt (res 2024-09-12 17:15:46.533 | INFO | bioimageio.core._resource_tests:_test_model_inference:147 - starting 'Reproduce test outputs from test inputs' /srv/conda/envs/notebook/lib/python3.10/site-packages/bioimageio/core/_resource_tests.py:407: UserWarning: The argument `decimal` has been depricated in favour of `relative_tolerance` and `absolute_tolerance`, with different validation logic, using `numpy.testing.assert_allclose, see 'https://numpy.org/doc/stable/reference/generated/numpy.testing.assert_allclose.html'. Passing a value for `decimal` will cause validation to revert to the old behaviour. warnings.warn( 2024-09-12 17:15:46.540 | INFO | bioimageio.core._resource_tests:_test_model_inference_parametrized:242 - Testing inference with 6 different input tensor sizes 2024-09-12 17:15:46.544 | INFO | bioimageio.core._resource_tests:_test_model_inference:147 - starting 'Reproduce test outputs from test inputs' 2024-09-12 17:15:46.555 | INFO | bioimageio.core._resource_tests:_test_model_inference_parametrized:242 - Testing inference with 6 different input tensor sizes 2024-09-12 17:15:46.557 | INFO | bioimageio.core._resource_tests:_test_model_inference:147 - starting 'Reproduce test outputs from test inputs' 2024-09-12 17:15:46.566 | INFO | bioimageio.core._resource_tests:_test_model_inference_parametrized:242 - Testing inference with 6 different input tensor sizes❌ bioimageio validation failed
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
source https://uk1s3.embassy.ebi.ac.uk/public-datasets/bioimage.…
format version model 0.5.3
bioimageio.spec 0.5.3.2
bioimageio.core 0.6.8
❓ location detail
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
✔️ initialized ModelDescr
to describe model 0.5.3
✔️ bioimageio.spec format
validation model 0.5.3
🔍 context.perform_io_che… True
🔍 context.root https://uk1s3.embassy.e…
🔍 context.known_files.ze… 767f2c3a50e36365c30b9e4…
🔍 context.known_files.te… c29bd6e16e3f7856217b407…
🔍 context.known_files.sa… a24b3c708b6ca6825494eb7…
🔍 context.known_files.te… 510181f38930e59e4fd8ecc…
🔍 context.known_files.sa… e8f99aabe8405427f515eba…
🔍 context.known_files.we… df913b85947f5132bcdaf81…
🔍 context.known_files.un… 7f5b15948e8e2c91f78dcff…
🔍 context.known_files.we… 608f52cd7f5119f7a7b8272…
🔍 context.known_files.we… 8410950508655a300793b38…
🔍 context.warning_level error
⚠ documentation No '# Validation'
(sub)section found in
documentation.md.
❌ Reproduce test outputs
from test inputs
❌ weights.onnx The 'onnx' model adapter
could not be created in
this environment:
ImportError(failed to
import onnxruntime: No
module named
'onnxruntime').
| | | �]8;id=839366;_resource_tests.py�_resource_tests.py�]8;;�,
line 161, in _test_model_inference with create_prediction_pipeline(
| | | |
�]8;id=688587;_prediction_pipeline.py�_prediction_pipeline.py�]8;;�, line 333, in create_prediction_pipeline
model_adapter = model_adapter or create_model_adapter( | | |
| �]8;id=670151;_model_adapter.py�_model_adapter.py�]8;;�, line 138, in create raise ValueError(
| | | |
| | ❌ | | Run onnx inference
for parametrized inputs
| | ❌ | weights.onnx | The 'onnx' model
adapter could not be created in this environment: ImportError(failed to import
onnxruntime: No module named 'onnxruntime').
| | | �]8;id=227383;_resource_tests.py�_resource_tests.py�]8;;�,
line 296, in _test_model_inference_parametrized with
create_prediction_pipeline( | | |
| �]8;id=394538;_prediction_pipeline.py�_prediction_pipeline.py�]8;;�, line 333, in create_prediction_pipeline
model_adapter = model_adapter or create_model_adapter( | | |
| �]8;id=235797;_model_adapter.py�_model_adapter.py�]8;;�, line 138, in create raise ValueError(
| | | |
| | ❌ | | Reproduce test
outputs from test inputs
| | ❌ | weights.pytorch_state_dict | The
'pytorch_state_dict' model adapter could not be created in this environment:
ImportError(failed to import torch: No module named 'torch').
| | | �]8;id=608868;_resource_tests.py�_resource_tests.py�]8;;�,
line 161, in _test_model_inference with create_prediction_pipeline(
| | | |
�]8;id=593174;_prediction_pipeline.py�_prediction_pipeline.py�]8;;�, line 333, in create_prediction_pipeline
model_adapter = model_adapter or create_model_adapter( | | |
| �]8;id=396184;_model_adapter.py�_model_adapter.py�]8;;�, line 138, in create raise ValueError(
| | | |
| | ❌ | | Run
pytorch_state_dict inference for parametrized inputs
| | ❌ | weights.pytorch_state_dict | The
'pytorch_state_dict' model adapter could not be created in this environment:
ImportError(failed to import torch: No module named 'torch').
| | | �]8;id=13371;_resource_tests.py�_resource_tests.py�]8;;�,
line 296, in _test_model_inference_parametrized with
create_prediction_pipeline( | | |
| �]8;id=137629;_prediction_pipeline.py�_prediction_pipeline.py�]8;;�, line 333, in create_prediction_pipeline
model_adapter = model_adapter or create_model_adapter( | | |
| �]8;id=975301;_model_adapter.py�_model_adapter.py�]8;;�, line 138, in create raise ValueError(
| | | |
| | ❌ | | Reproduce test
outputs from test inputs
| | ❌ | weights.torchscript | The 'torchscript'
model adapter could not be created in this environment: ImportError(failed to
import torch: No module named 'torch').
| | | �]8;id=429218;_resource_tests.py�_resource_tests.py�]8;;�,
line 161, in _test_model_inference with create_prediction_pipeline(
| | | |
�]8;id=163549;_prediction_pipeline.py�_prediction_pipeline.py�]8;;�, line 333, in create_prediction_pipeline
model_adapter = model_adapter or create_model_adapter( | | |
| �]8;id=43384;_model_adapter.py�_model_adapter.py�]8;;�, line 138, in create raise ValueError(
| | | |
| | ❌ | | Run torchscript
inference for parametrized inputs
| | ❌ | weights.torchscript | The 'torchscript'
model adapter could not be created in this environment: ImportError(failed to
import torch: No module named 'torch').
| | | �]8;id=935828;_resource_tests.py�_resource_tests.py�]8;;�,
line 296, in _test_model_inference_parametrized with
create_prediction_pipeline( | | |
| �]8;id=672681;_prediction_pipeline.py�_prediction_pipeline.py�]8;;�, line 333, in create_prediction_pipeline
model_adapter = model_adapter or create_model_adapter( | | |
| �]8;id=379847;_model_adapter.py�_model_adapter.py�]8;;�, line 138, in create raise ValueError(
| | | |
The text was updated successfully, but these errors were encountered: