Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Previously working code has stopped to work with import error #375

Closed
jhennies opened this issue Apr 16, 2024 · 8 comments · May be fixed by #377
Closed

Previously working code has stopped to work with import error #375

jhennies opened this issue Apr 16, 2024 · 8 comments · May be fixed by #377

Comments

@jhennies
Copy link
Contributor

Hi all,

Due to my stuff not working anymore complaining about some pytorch internal issue and core-dumps, I re-installed my conda environment but still my previously working code won't work due to some import error (code and error below).

Is this a me-problem or might something be broken on the bioimage-io side?

This is what I have installed:

bioimageio.core           0.5.11             pyhd8ed1ab_0    conda-forge
bioimageio.spec           0.5.1              pyhd8ed1ab_0    conda-forge

Code:

def run_cebra_net(
        input_data
):

    # Set up the model
    try:
        rdf_path = os.path.join(
            get_repo_path(), 'models', 'cebranet-cellular-membranes-in-volume-sem_torchscript.zip'
        )
        print(f'rdf_path = {rdf_path}')
        model_resource = bioimageio.core.load_resource_description(rdf_path)
    except TypeError:
        rdf_doi = "10.5281/zenodo.7274275"
        model_resource = bioimageio.core.load_resource_description(rdf_doi)

    # Prepare the data
    input_array = xr.DataArray(input_data[None, None, :], dims=tuple(model_resource.inputs[0].axes))

    # Set up the prediction pipeline
    devices = None
    weight_format = None
    prediction_pipeline = bioimageio.core.create_prediction_pipeline(
        model_resource, devices=devices, weight_format=weight_format
    )

    # Make sure the input and output shapes match
    prediction_pipeline.input_specs[0].shape = prediction_pipeline.output_specs[0].shape
    tiling = True

    # Predict with tiling
    result = bioimageio.core.predict_with_tiling(prediction_pipeline, input_array, tiling=tiling, verbose=True)[0]

    return (result[0, 0, :] * 255).astype('uint8')

Error:

/home/julian/miniconda3/envs/cebra-em-all-env/lib/python3.9/site-packages/networkx/utils/backends.py:135: RuntimeWarning: networkx backend defined more than once: nx-loopback
  backends.update(_get_backends("networkx.backends"))
Traceback (most recent call last):
  File "/media/julian/Data/tmp/test_cebra_em/proj/.snakemake/scripts/tmpehogir6d.run_task.py", line 17, in <module>
    from cebra_em_core.bioimageio.cebra_net import run_cebra_net
  File "/media/julian/Data/src/github/jhennies/CebraEM/cebra-em-core/cebra_em_core/bioimageio/cebra_net.py", line 2, in <module>
    import bioimageio.core
  File "/home/julian/miniconda3/envs/cebra-em-all-env/lib/python3.9/site-packages/bioimageio/core/__init__.py", line 6, in <module>
    from .resource_io import (
  File "/home/julian/miniconda3/envs/cebra-em-all-env/lib/python3.9/site-packages/bioimageio/core/resource_io/__init__.py", line 2, in <module>
    from .io_ import (
  File "/home/julian/miniconda3/envs/cebra-em-all-env/lib/python3.9/site-packages/bioimageio/core/resource_io/io_.py", line 11, in <module>
    from bioimageio.core.resource_io.nodes import ResourceDescription
  File "/home/julian/miniconda3/envs/cebra-em-all-env/lib/python3.9/site-packages/bioimageio/core/resource_io/nodes.py", line 9, in <module>
    from bioimageio.spec.model import raw_nodes as model_raw_nodes
ImportError: cannot import name 'raw_nodes' from 'bioimageio.spec.model' (/home/julian/miniconda3/envs/cebra-em-all-env/lib/python3.9/site-packages/bioimageio/spec/model/__init__.py)
@FynnBe
Copy link
Member

FynnBe commented Apr 16, 2024

How did you install spec and core? Those versions are not compatible

@jhennies
Copy link
Contributor Author

jhennies commented Apr 16, 2024 •
edited
Loading

conda create -n bioimageio-test-env -c conda-forge bioimageio.core
(I also had other packages in the create command, but this also yields the versions specified above)

@FynnBe
Copy link
Member

FynnBe commented Apr 16, 2024

hmm.. really good to know that the pin we set here does not make it into the conda-recipe...

@jhennies
Copy link
Contributor Author

So in other words, I would need to downgrade the specs to 0.4.9?

@FynnBe
Copy link
Member

FynnBe commented Apr 16, 2024

currently yes, the new core release is coming very soon though, so maybe wait just a few days if you can

@jhennies
Copy link
Contributor Author

Ok, that gets rid of the import error. But I'm back to a core dump, looking like this:

prediction with tiling:   0%|                                                                                                                                                                                                                                       | 0/1000 [00:00<?, ?it/s]terminate called after throwing an instance of 'c10::Error'
  what():  isTuple() INTERNAL ASSERT FAILED at "/pytorch/aten/src/ATen/core/ivalue_inl.h":931, please report a bug to PyTorch. Expected Tuple but got String
Exception raised from toTuple at /pytorch/aten/src/ATen/core/ivalue_inl.h:931 (most recent call first):
frame #0: c10::Error::Error(c10::SourceLocation, std::string) + 0x42 (0x7f2ed5dc88b2 in /home/julian/miniconda3/envs/cebra-em-all-env/lib/python3.9/site-packages/torch/lib/libc10.so)
frame #1: <unknown function> + 0x2f850b0 (0x7f2f6333a0b0 in /home/julian/miniconda3/envs/cebra-em-all-env/lib/python3.9/site-packages/torch/lib/libtorch_cpu.so)
frame #2: <unknown function> + 0x2f873a5 (0x7f2f6333c3a5 in /home/julian/miniconda3/envs/cebra-em-all-env/lib/python3.9/site-packages/torch/lib/libtorch_cpu.so)
frame #3: torch::jit::SourceRange::highlight(std::ostream&) const + 0x38 (0x7f2f60fc4d68 in /home/julian/miniconda3/envs/cebra-em-all-env/lib/python3.9/site-packages/torch/lib/libtorch_cpu.so)
frame #4: torch::jit::ErrorReport::what() const + 0x2c5 (0x7f2f60fa42d5 in /home/julian/miniconda3/envs/cebra-em-all-env/lib/python3.9/site-packages/torch/lib/libtorch_cpu.so)
frame #5: <unknown function> + 0x2b11ce (0x7f2f7137e1ce in /home/julian/miniconda3/envs/cebra-em-all-env/lib/python3.9/site-packages/torch/lib/libtorch_python.so)
frame #6: <unknown function> + 0x2c4fa6 (0x7f2f71391fa6 in /home/julian/miniconda3/envs/cebra-em-all-env/lib/python3.9/site-packages/torch/lib/libtorch_python.so)
frame #7: <unknown function> + 0x14e6b6 (0x55caf49286b6 in /home/julian/miniconda3/envs/cebra-em-all-env/bin/python3.9)
frame #8: _PyObject_MakeTpCall + 0x2ec (0x55caf49117ac in /home/julian/miniconda3/envs/cebra-em-all-env/bin/python3.9)
frame #9: _PyEval_EvalFrameDefault + 0x4fa0 (0x55caf490df10 in /home/julian/miniconda3/envs/cebra-em-all-env/bin/python3.9)
frame #10: <unknown function> + 0x12e184 (0x55caf4908184 in /home/julian/miniconda3/envs/cebra-em-all-env/bin/python3.9)
frame #11: _PyFunction_Vectorcall + 0xd9 (0x55caf4919559 in /home/julian/miniconda3/envs/cebra-em-all-env/bin/python3.9)
frame #12: _PyEval_EvalFrameDefault + 0x4abe (0x55caf490da2e in /home/julian/miniconda3/envs/cebra-em-all-env/bin/python3.9)
frame #13: <unknown function> + 0x12e184 (0x55caf4908184 in /home/julian/miniconda3/envs/cebra-em-all-env/bin/python3.9)
frame #14: <unknown function> + 0x14c9fb (0x55caf49269fb in /home/julian/miniconda3/envs/cebra-em-all-env/bin/python3.9)
frame #15: _PyEval_EvalFrameDefault + 0x11f2 (0x55caf490a162 in /home/julian/miniconda3/envs/cebra-em-all-env/bin/python3.9)
frame #16: <unknown function> + 0x12e184 (0x55caf4908184 in /home/julian/miniconda3/envs/cebra-em-all-env/bin/python3.9)
frame #17: _PyFunction_Vectorcall + 0xd9 (0x55caf4919559 in /home/julian/miniconda3/envs/cebra-em-all-env/bin/python3.9)
frame #18: _PyEval_EvalFrameDefault + 0x696 (0x55caf4909606 in /home/julian/miniconda3/envs/cebra-em-all-env/bin/python3.9)
frame #19: <unknown function> + 0x12e184 (0x55caf4908184 in /home/julian/miniconda3/envs/cebra-em-all-env/bin/python3.9)
frame #20: _PyFunction_Vectorcall + 0xd9 (0x55caf4919559 in /home/julian/miniconda3/envs/cebra-em-all-env/bin/python3.9)
frame #21: <unknown function> + 0x14cadc (0x55caf4926adc in /home/julian/miniconda3/envs/cebra-em-all-env/bin/python3.9)
frame #22: _PyEval_EvalFrameDefault + 0x45a6 (0x55caf490d516 in /home/julian/miniconda3/envs/cebra-em-all-env/bin/python3.9)
frame #23: <unknown function> + 0x12e184 (0x55caf4908184 in /home/julian/miniconda3/envs/cebra-em-all-env/bin/python3.9)
frame #24: _PyFunction_Vectorcall + 0xd9 (0x55caf4919559 in /home/julian/miniconda3/envs/cebra-em-all-env/bin/python3.9)
frame #25: <unknown function> + 0x14cadc (0x55caf4926adc in /home/julian/miniconda3/envs/cebra-em-all-env/bin/python3.9)
frame #26: _PyEval_EvalFrameDefault + 0x45a6 (0x55caf490d516 in /home/julian/miniconda3/envs/cebra-em-all-env/bin/python3.9)
frame #27: <unknown function> + 0x12e184 (0x55caf4908184 in /home/julian/miniconda3/envs/cebra-em-all-env/bin/python3.9)
frame #28: _PyFunction_Vectorcall + 0xd9 (0x55caf4919559 in /home/julian/miniconda3/envs/cebra-em-all-env/bin/python3.9)
frame #29: <unknown function> + 0x14cadc (0x55caf4926adc in /home/julian/miniconda3/envs/cebra-em-all-env/bin/python3.9)
frame #30: _PyEval_EvalFrameDefault + 0x45a6 (0x55caf490d516 in /home/julian/miniconda3/envs/cebra-em-all-env/bin/python3.9)
frame #31: <unknown function> + 0x13f802 (0x55caf4919802 in /home/julian/miniconda3/envs/cebra-em-all-env/bin/python3.9)
frame #32: _PyEval_EvalFrameDefault + 0x3e2 (0x55caf4909352 in /home/julian/miniconda3/envs/cebra-em-all-env/bin/python3.9)
frame #33: <unknown function> + 0x12e184 (0x55caf4908184 in /home/julian/miniconda3/envs/cebra-em-all-env/bin/python3.9)
frame #34: _PyFunction_Vectorcall + 0xd9 (0x55caf4919559 in /home/julian/miniconda3/envs/cebra-em-all-env/bin/python3.9)
frame #35: _PyEval_EvalFrameDefault + 0x3e2 (0x55caf4909352 in /home/julian/miniconda3/envs/cebra-em-all-env/bin/python3.9)
frame #36: <unknown function> + 0x12e184 (0x55caf4908184 in /home/julian/miniconda3/envs/cebra-em-all-env/bin/python3.9)
frame #37: _PyFunction_Vectorcall + 0xd9 (0x55caf4919559 in /home/julian/miniconda3/envs/cebra-em-all-env/bin/python3.9)
frame #38: _PyEval_EvalFrameDefault + 0x11f2 (0x55caf490a162 in /home/julian/miniconda3/envs/cebra-em-all-env/bin/python3.9)
frame #39: <unknown function> + 0x12e184 (0x55caf4908184 in /home/julian/miniconda3/envs/cebra-em-all-env/bin/python3.9)
frame #40: _PyFunction_Vectorcall + 0xd9 (0x55caf4919559 in /home/julian/miniconda3/envs/cebra-em-all-env/bin/python3.9)
frame #41: _PyEval_EvalFrameDefault + 0x11f2 (0x55caf490a162 in /home/julian/miniconda3/envs/cebra-em-all-env/bin/python3.9)
frame #42: <unknown function> + 0x13f802 (0x55caf4919802 in /home/julian/miniconda3/envs/cebra-em-all-env/bin/python3.9)
frame #43: _PyEval_EvalFrameDefault + 0x3e2 (0x55caf4909352 in /home/julian/miniconda3/envs/cebra-em-all-env/bin/python3.9)
frame #44: <unknown function> + 0x12e184 (0x55caf4908184 in /home/julian/miniconda3/envs/cebra-em-all-env/bin/python3.9)
frame #45: _PyFunction_Vectorcall + 0xd9 (0x55caf4919559 in /home/julian/miniconda3/envs/cebra-em-all-env/bin/python3.9)
frame #46: _PyEval_EvalFrameDefault + 0x3e2 (0x55caf4909352 in /home/julian/miniconda3/envs/cebra-em-all-env/bin/python3.9)
frame #47: <unknown function> + 0x12e184 (0x55caf4908184 in /home/julian/miniconda3/envs/cebra-em-all-env/bin/python3.9)
frame #48: _PyFunction_Vectorcall + 0xd9 (0x55caf4919559 in /home/julian/miniconda3/envs/cebra-em-all-env/bin/python3.9)
frame #49: _PyEval_EvalFrameDefault + 0x11f2 (0x55caf490a162 in /home/julian/miniconda3/envs/cebra-em-all-env/bin/python3.9)
frame #50: <unknown function> + 0x12e184 (0x55caf4908184 in /home/julian/miniconda3/envs/cebra-em-all-env/bin/python3.9)
frame #51: _PyEval_EvalCodeWithName + 0x48 (0x55caf4907e58 in /home/julian/miniconda3/envs/cebra-em-all-env/bin/python3.9)
frame #52: PyEval_EvalCodeEx + 0x39 (0x55caf4907e09 in /home/julian/miniconda3/envs/cebra-em-all-env/bin/python3.9)
frame #53: PyEval_EvalCode + 0x1b (0x55caf49b54ab in /home/julian/miniconda3/envs/cebra-em-all-env/bin/python3.9)
frame #54: <unknown function> + 0x20887a (0x55caf49e287a in /home/julian/miniconda3/envs/cebra-em-all-env/bin/python3.9)
frame #55: <unknown function> + 0x204c03 (0x55caf49dec03 in /home/julian/miniconda3/envs/cebra-em-all-env/bin/python3.9)
frame #56: <unknown function> + 0x98175 (0x55caf4872175 in /home/julian/miniconda3/envs/cebra-em-all-env/bin/python3.9)
frame #57: PyRun_SimpleFileExFlags + 0x1b1 (0x55caf49d8841 in /home/julian/miniconda3/envs/cebra-em-all-env/bin/python3.9)
frame #58: Py_RunMain + 0x38d (0x55caf49d5c6d in /home/julian/miniconda3/envs/cebra-em-all-env/bin/python3.9)
frame #59: Py_BytesMain + 0x37 (0x55caf49a92e7 in /home/julian/miniconda3/envs/cebra-em-all-env/bin/python3.9)
frame #60: __libc_start_main + 0xf3 (0x7f2f88108083 in /lib/x86_64-linux-gnu/libc.so.6)
frame #61: <unknown function> + 0x1cf1e1 (0x55caf49a91e1 in /home/julian/miniconda3/envs/cebra-em-all-env/bin/python3.9)

/usr/bin/bash: line 1: 26594 Aborted                 (core dumped) /home/julian/miniconda3/envs/cebra-em-all-env/bin/python3.9 /media/julian/Data/tmp/test_cebra_em/proj/.snakemake/scripts/tmp8ml04ipe.run_task.py

Have you seen this before? Or any idea where I might need to look into?

@FynnBe
Copy link
Member

FynnBe commented Apr 16, 2024

looks like you're trying to run a model traced on GPU on CPU (or vice versa??) (libtorch_cpu.so)
makes sure inputs and model are on the same device and try GPU as well as CPU.. maybe that will get you somewhere

@jhennies
Copy link
Contributor Author

Thanks @FynnBe, it was some pytorch installation issue, so a me-problem.
From my side the issue is solved, meaning everything works for me (respective the import error). Do you want to keep the issue open as it requires manually specifying the spec version?

@FynnBe FynnBe mentioned this issue Apr 23, 2024
Open
@FynnBe FynnBe closed this as completed May 24, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

bump post bioimage-io/core-bioimage-io-python
2 participants
@jhennies @FynnBe