Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

ERROR: model not working anymore #52

Open
pr4deepr opened this issue Nov 18, 2024 · 47 comments
Open

ERROR: model not working anymore #52

pr4deepr opened this issue Nov 18, 2024 · 47 comments

Comments

@pr4deepr
Copy link

pr4deepr commented Nov 18, 2024

Hi
After the recent update, my deepimageJ bundled model as part of a software package is not working anymore.

You can download the model here:
https://zenodo.org/records/10460434 and get the file: 2D_enteric_ganglia_model_QA.zip

The model is under:
2D_Ganglia_RGB_v2.bioimage.io.model.zip

I get the following error when I run it on the test image:

[SERVICE-0] {"task":"e6be41f1-3206-4ac9-b120-51d6feb9ef13","requestType":"EXECUTE","inputs":{"modelFolder":"C:\\Users\\Pradeep\\Fiji.app\\models\\2D_Ganglia_RGB_v2.bioimage.io.model"},"script":"loadModel"}
[SERVICE-0] {"responseType":"LAUNCH","task":"e6be41f1-3206-4ac9-b120-51d6feb9ef13"}
[WORKER-0] 2024-11-18 13:47:52.128680: W tensorflow/stream_executor/platform/default/dso_loader.cc:55] Could not load dynamic library 'cudart64_100.dll'; dlerror: cudart64_100.dll not found
[WORKER-0] 2024-11-18 13:47:52.128719: I tensorflow/stream_executor/cuda/cudart_stub.cc:29] Ignore above cudart dlerror if you do not have a GPU set up on your machine.
[WORKER-0] 2024-11-18 13:47:52.168193: I tensorflow/cc/saved_model/reader.cc:31] Reading SavedModel from: C:\Users\Pradeep\Fiji.app\models\2D_Ganglia_RGB_v2.bioimage.io.model
[WORKER-0] 2024-11-18 13:47:52.288516: I tensorflow/cc/saved_model/reader.cc:54] Reading meta graph with tags { serve }
[WORKER-0] 2024-11-18 13:47:52.560161: I tensorflow/core/platform/cpu_feature_guard.cc:142] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2
[WORKER-0] 2024-11-18 13:47:52.563946: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library nvcuda.dll
[WORKER-0] 2024-11-18 13:47:52.596451: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1618] Found device 0 with properties: 
[WORKER-0] name: NVIDIA GeForce RTX 3080 major: 8 minor: 6 memoryClockRate(GHz): 1.755
[WORKER-0] pciBusID: 0000:2d:00.0
[WORKER-0] 2024-11-18 13:47:52.598348: W tensorflow/stream_executor/platform/default/dso_loader.cc:55] Could not load dynamic library 'cudart64_100.dll'; dlerror: cudart64_100.dll not found
[WORKER-0] 2024-11-18 13:47:52.599938: W tensorflow/stream_executor/platform/default/dso_loader.cc:55] Could not load dynamic library 'cublas64_100.dll'; dlerror: cublas64_100.dll not found
[WORKER-0] 2024-11-18 13:47:52.601426: W tensorflow/stream_executor/platform/default/dso_loader.cc:55] Could not load dynamic library 'cufft64_100.dll'; dlerror: cufft64_100.dll not found
[WORKER-0] 2024-11-18 13:47:52.602857: W tensorflow/stream_executor/platform/default/dso_loader.cc:55] Could not load dynamic library 'curand64_100.dll'; dlerror: curand64_100.dll not found
[WORKER-0] 2024-11-18 13:47:52.604167: W tensorflow/stream_executor/platform/default/dso_loader.cc:55] Could not load dynamic library 'cusolver64_100.dll'; dlerror: cusolver64_100.dll not found
[WORKER-0] 2024-11-18 13:47:52.605352: W tensorflow/stream_executor/platform/default/dso_loader.cc:55] Could not load dynamic library 'cusparse64_100.dll'; dlerror: cusparse64_100.dll not found
[WORKER-0] 2024-11-18 13:47:52.606367: W tensorflow/stream_executor/platform/default/dso_loader.cc:55] Could not load dynamic library 'cudnn64_7.dll'; dlerror: cudnn64_7.dll not found
[WORKER-0] 2024-11-18 13:47:52.606390: W tensorflow/core/common_runtime/gpu/gpu_device.cc:1641] Cannot dlopen some GPU libraries. Please make sure the missing libraries mentioned above are installed properly if you would like to use GPU. Follow the guide at https://www.tensorflow.org/install/gpu for how to download and setup the required libraries for your platform.
[WORKER-0] Skipping registering GPU devices...
[WORKER-0] 2024-11-18 13:47:52.690799: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1159] Device interconnect StreamExecutor with strength 1 edge matrix:
[WORKER-0] 2024-11-18 13:47:52.690840: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1165]      0 
[WORKER-0] 2024-11-18 13:47:52.690847: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1178] 0:   N 
[WORKER-0] 2024-11-18 13:47:53.691620: I tensorflow/cc/saved_model/loader.cc:202] Restoring SavedModel bundle.
[WORKER-0] 2024-11-18 13:47:57.102229: I tensorflow/cc/saved_model/loader.cc:311] SavedModel load for tags { serve }; Status: success. Took 4934026 microseconds.
[SERVICE-0] {"responseType":"COMPLETION","task":"e6be41f1-3206-4ac9-b120-51d6feb9ef13"}
java.lang.IllegalArgumentException: Tiling cannot happen accross the channel dimension. The tile number of channels (3) must be the same as the image number of channels (1).
	at io.bioimage.modelrunner.bioimageio.tiling.TileMaker.validateTileVsImageChannel(TileMaker.java:209)
	at io.bioimage.modelrunner.bioimageio.tiling.TileMaker.validate(TileMaker.java:71)
	at io.bioimage.modelrunner.bioimageio.tiling.TileMaker.<init>(TileMaker.java:48)
	at io.bioimage.modelrunner.bioimageio.tiling.TileMaker.build(TileMaker.java:62)
	at io.bioimage.modelrunner.model.Model.runBMZ(Model.java:613)
	at io.bioimage.modelrunner.model.Model.runBMZ(Model.java:575)
	at deepimagej.Runner.run(Runner.java:138)
	at deepimagej.gui.Gui.lambda$runModel$17(Gui.java:258)
	at java.lang.Thread.run(Thread.java:750)

@pr4deepr
Copy link
Author

I am also planning to release a new model soon, let do let me know what I need to change in the rdf.yaml file

@pr4deepr
Copy link
Author

So, I restarted Fiji and I ran it again..

image

Its like the axes got flipped..

[SERVICE-0] {"task":"dc1e2e28-782e-4084-a7df-6f1b0611e185","requestType":"EXECUTE","inputs":{"modelFolder":"C:\\Users\\Pradeep\\Fiji.app\\models\\2D_Ganglia_RGB_v2.bioimage.io.model"},"script":"loadModel"}
[SERVICE-0] {"responseType":"LAUNCH","task":"dc1e2e28-782e-4084-a7df-6f1b0611e185"}
[WORKER-0] 2024-11-18 14:01:39.404921: W tensorflow/stream_executor/platform/default/dso_loader.cc:55] Could not load dynamic library 'cudart64_100.dll'; dlerror: cudart64_100.dll not found
[WORKER-0] 2024-11-18 14:01:39.404962: I tensorflow/stream_executor/cuda/cudart_stub.cc:29] Ignore above cudart dlerror if you do not have a GPU set up on your machine.
[WORKER-0] 2024-11-18 14:01:39.446292: I tensorflow/cc/saved_model/reader.cc:31] Reading SavedModel from: C:\Users\Pradeep\Fiji.app\models\2D_Ganglia_RGB_v2.bioimage.io.model
[WORKER-0] 2024-11-18 14:01:39.567890: I tensorflow/cc/saved_model/reader.cc:54] Reading meta graph with tags { serve }
[WORKER-0] 2024-11-18 14:01:39.839230: I tensorflow/core/platform/cpu_feature_guard.cc:142] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2
[WORKER-0] 2024-11-18 14:01:39.843172: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library nvcuda.dll
[WORKER-0] 2024-11-18 14:01:39.876067: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1618] Found device 0 with properties: 
[WORKER-0] name: NVIDIA GeForce RTX 3080 major: 8 minor: 6 memoryClockRate(GHz): 1.755
[WORKER-0] pciBusID: 0000:2d:00.0
[WORKER-0] 2024-11-18 14:01:39.877382: W tensorflow/stream_executor/platform/default/dso_loader.cc:55] Could not load dynamic library 'cudart64_100.dll'; dlerror: cudart64_100.dll not found
[WORKER-0] 2024-11-18 14:01:39.878382: W tensorflow/stream_executor/platform/default/dso_loader.cc:55] Could not load dynamic library 'cublas64_100.dll'; dlerror: cublas64_100.dll not found
[WORKER-0] 2024-11-18 14:01:39.879398: W tensorflow/stream_executor/platform/default/dso_loader.cc:55] Could not load dynamic library 'cufft64_100.dll'; dlerror: cufft64_100.dll not found
[WORKER-0] 2024-11-18 14:01:39.880339: W tensorflow/stream_executor/platform/default/dso_loader.cc:55] Could not load dynamic library 'curand64_100.dll'; dlerror: curand64_100.dll not found
[WORKER-0] 2024-11-18 14:01:39.881294: W tensorflow/stream_executor/platform/default/dso_loader.cc:55] Could not load dynamic library 'cusolver64_100.dll'; dlerror: cusolver64_100.dll not found
[WORKER-0] 2024-11-18 14:01:39.882213: W tensorflow/stream_executor/platform/default/dso_loader.cc:55] Could not load dynamic library 'cusparse64_100.dll'; dlerror: cusparse64_100.dll not found
[WORKER-0] 2024-11-18 14:01:39.883107: W tensorflow/stream_executor/platform/default/dso_loader.cc:55] Could not load dynamic library 'cudnn64_7.dll'; dlerror: cudnn64_7.dll not found
[WORKER-0] 2024-11-18 14:01:39.883127: W tensorflow/core/common_runtime/gpu/gpu_device.cc:1641] Cannot dlopen some GPU libraries. Please make sure the missing libraries mentioned above are installed properly if you would like to use GPU. Follow the guide at https://www.tensorflow.org/install/gpu for how to download and setup the required libraries for your platform.
[WORKER-0] Skipping registering GPU devices...
[WORKER-0] 2024-11-18 14:01:39.965190: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1159] Device interconnect StreamExecutor with strength 1 edge matrix:
[WORKER-0] 2024-11-18 14:01:39.965224: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1165]      0 
[WORKER-0] 2024-11-18 14:01:39.965233: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1178] 0:   N 
[WORKER-0] 2024-11-18 14:01:41.003185: I tensorflow/cc/saved_model/loader.cc:202] Restoring SavedModel bundle.
[WORKER-0] 2024-11-18 14:01:44.712470: I tensorflow/cc/saved_model/loader.cc:311] SavedModel load for tags { serve }; Status: success. Took 5266171 microseconds.
[SERVICE-0] {"responseType":"COMPLETION","task":"dc1e2e28-782e-4084-a7df-6f1b0611e185"}
[SERVICE-0] {"task":"492ab921-2f18-4fba-9814-6d724d27eb52","requestType":"EXECUTE","inputs":{"outputs":["{\"memoryName\":\"Local\\\\4cab69a1-908c-43af-b1dc-3c9720a2af9e\",\"isInput\":false,\"shape\":[1,768,768,3],\"name\":\"output\",\"dtype\":\"float32\"}"],"inputs":["{\"memoryName\":\"Local\\\\56e13aa8-51c5-4643-9d11-57f85161f9af\",\"shape\":[1,768,768,3],\"isInput\":true,\"name\":\"input\",\"dtype\":\"float32\"}"]},"script":"inference"}
[SERVICE-0] {"responseType":"LAUNCH","task":"492ab921-2f18-4fba-9814-6d724d27eb52"}
[SERVICE-0] {"responseType":"COMPLETION","task":"492ab921-2f18-4fba-9814-6d724d27eb52"}
[SERVICE-0] {"task":"e2e7d6f0-9b2e-4cfc-9059-7e38d883252c","requestType":"EXECUTE","inputs":{"outputs":["{\"memoryName\":\"Local\\\\c6787ef6-b53e-4105-a7e1-7066167fe489\",\"isInput\":false,\"shape\":[1,768,768,3],\"name\":\"output\",\"dtype\":\"float32\"}"],"inputs":["{\"memoryName\":\"Local\\\\c7aceaf6-200d-4f18-a8e0-64338667023b\",\"shape\":[1,768,768,3],\"isInput\":true,\"name\":\"input\",\"dtype\":\"float32\"}"]},"script":"inference"}
[SERVICE-0] {"responseType":"LAUNCH","task":"e2e7d6f0-9b2e-4cfc-9059-7e38d883252c"}
[SERVICE-0] {"responseType":"COMPLETION","task":"e2e7d6f0-9b2e-4cfc-9059-7e38d883252c"}
[SERVICE-0] {"task":"590b9f28-b16b-4afb-8069-b33abc7f5323","requestType":"EXECUTE","inputs":{"outputs":["{\"memoryName\":\"Local\\\\65ffa681-9e5b-40f7-84c2-1d0b363bb87a\",\"isInput\":false,\"shape\":[1,768,768,3],\"name\":\"output\",\"dtype\":\"float32\"}"],"inputs":["{\"memoryName\":\"Local\\\\a9c0c8c7-b3a9-4750-b959-b131b851c119\",\"shape\":[1,768,768,3],\"isInput\":true,\"name\":\"input\",\"dtype\":\"float32\"}"]},"script":"inference"}
[SERVICE-0] {"responseType":"LAUNCH","task":"590b9f28-b16b-4afb-8069-b33abc7f5323"}
[SERVICE-0] {"responseType":"COMPLETION","task":"590b9f28-b16b-4afb-8069-b33abc7f5323"}
[SERVICE-0] {"task":"ed06d97a-41e0-4e44-a87e-364343c5e270","requestType":"EXECUTE","inputs":{"outputs":["{\"memoryName\":\"Local\\\\95c51b15-9867-4a68-bd84-ae0db80c3a46\",\"isInput\":false,\"shape\":[1,768,768,3],\"name\":\"output\",\"dtype\":\"float32\"}"],"inputs":["{\"memoryName\":\"Local\\\\581184bd-c685-452d-a149-57002c618855\",\"shape\":[1,768,768,3],\"isInput\":true,\"name\":\"input\",\"dtype\":\"float32\"}"]},"script":"inference"}
[SERVICE-0] {"responseType":"LAUNCH","task":"ed06d97a-41e0-4e44-a87e-364343c5e270"}
[SERVICE-0] {"responseType":"COMPLETION","task":"ed06d97a-41e0-4e44-a87e-364343c5e270"}

@carlosuc3m
Copy link
Collaborator

HEllo @pr4deepr
First of all thank you for taking your time to make it work!

I hae downloaed the model at: https://zenodo.org/records/10460434/files/2D_enteric_ganglia_model_QA.zip?download=1

I have tried the model ´D_Ganglia_RGB_v2.bioimage.io.model` from that folder.

It did not work for you at first because the output of the model has only one channel, where as the rdf.yaml specifies 3.

Change the following:

outputs:
- axes: bxyc
  data_range: [0.0, 255.0]
  data_type: uint8
  name: output
  postprocessing:
  - kwargs: {threshold: 0.7364705882352942}
    name: binarize
  halo:
      - 0
      - 191
      - 191
      - 0
  shape:
      offset:
        - 0
        - 0
        - 0
        - 0
      reference_tensor: input
      scale:
        - 1
        - 1
        - 1
        - 3

by this:

outputs:
- axes: bxyc
  data_range: [0.0, 255.0]
  data_type: uint8
  name: output
  postprocessing:
  - kwargs: {threshold: 0.7364705882352942}
    name: binarize
  halo:
      - 0
      - 191
      - 191
      - 0
  shape:
      offset:
        - 0
        - 0
        - 0
        - 0.5 ## HERE
      reference_tensor: input
      scale:
        - 1
        - 1
        - 1
        - 0 ## HERE

Or this

outputs:
- axes: bxyc
  data_range: [0.0, 255.0]
  data_type: uint8
  name: output
  postprocessing:
  - kwargs: {threshold: 0.7364705882352942}
    name: binarize
  halo:
      - 0
      - 191
      - 191
      - 0
  shape: [1, 768, 768, 1]

With this change, the model works. And produces the same output as the test_output.npy.

@carlosuc3m
Copy link
Collaborator

@pr4deepr
Unfortunately, deepImageJ is getting rid of the packager, so it now depends on manually changing the fields of the rdf.yaml or using one the bioimage.io packagers that exist.

Sorry for this!
As you might have seen we are making big changes on deepImageJ. If you have any suggestion this is the time to do it!

REgards,
Carlos

@carlosuc3m
Copy link
Collaborator

HEllo again @pr4deepr
Sorry but it seems that the model does not produced the same output as expected. It might be due to pre or post-processing

@pr4deepr
Copy link
Author

As you might have seen we are making big changes on deepImageJ. If you have any suggestion this is the time to do it!

Having a packager of some sort is super handy.. Its quite overwhelming to figure out how to manually edit/create rdf.yaml file otherwise..
If not, atleast a webform of sorts where we can enter fields and export the yaml file will be super handy.

In the plugin it was really handy to be able to turn pre or post processing on or off. Is it possible to have like a developer or advanced setting?

Sorry but it seems that the model does not produced the same output as expected. It might be due to pre or post-processing
The preprocessing takes an RGB image, normalizes each channel independently, passes it onto the model for prediction.
The prediction still had 3 channels, and channel 1 had the prediction. So, the custom postprocessing script duplicates only channel 1 and applies a custom threshold.

@carlosuc3m
Copy link
Collaborator

Hello @pr4deepr
Okk thanks for your suggestions.
Both things were already on the table, so they should come in the near-medium future, hopefully with many more features!

Does the model work now as expected?

Carlos

@pr4deepr
Copy link
Author

Thanks for all the suggestions

Does the model work now as expected?

No, it doesn't. When I try:

outputs:

  • axes: bxyc
    data_range: [0.0, 255.0]
    data_type: uint8
    name: output
    postprocessing:
    • kwargs: {threshold: 0.7364705882352942}
      name: binarize
      halo:
      • 0
      • 191
      • 191
      • 0
        shape:
        offset:
        • 0
        • 0
        • 0
        • 0.5 ## HERE
          reference_tensor: input
          scale:
        • 1
        • 1
        • 1
        • 0 ## HE

Image on left is prediction and image on right is what I expect to get:

image

which is incorrect prediction.. Looks like a really low threshold was preformed?

WIth this change

outputs:

  • axes: bxyc
    data_range: [0.0, 255.0]
    data_type: uint8
    name: output
    postprocessing:
    • kwargs: {threshold: 0.7364705882352942}
      name: binarize
      halo:
      • 0
      • 191
      • 191
      • 0
        shape: [1, 768, 768, 1]

I get:

image

@carlosuc3m
Copy link
Collaborator

HEllo @pr4deepr
What about the pre-processing? is it correct? What is the expected preprocessing?
Regards,
Carlos

@carlosuc3m
Copy link
Collaborator

@pr4deepr
Hello I have tried the model with Tensorflow 1 in Python and it returns the same thing as in deepImageJ, which does not correspond to the sample output in .npy.

I have used the input .npy as the input image.

Maybe try it yourself with the keras model to see if the results are what you expect. If they are, there must have been an error with the conversion from keras to tf1, if not the keras model might be corrupted.
REgards,
Carlos

@abadgerw
Copy link

@carlosuc3m I am also running into the same channel error described above:

[SERVICE-2] {“task”:“0bae35a2-4113-4b8c-9568-8b6737ea9a8b”,“requestType”:“EXECUTE”,“inputs”:{“modelSource”:“/Applications/Fiji.app/models/InstanSeg_brightfield_nuclei/instanseg.pt”,“modelFolder”:“/Applications/Fiji.app/models/InstanSeg_brightfield_nuclei”},“script”:“loadModel”}
[SERVICE-2] {“responseType”:“LAUNCH”,“task”:“0bae35a2-4113-4b8c-9568-8b6737ea9a8b”}
[SERVICE-2] {“responseType”:“COMPLETION”,“task”:“0bae35a2-4113-4b8c-9568-8b6737ea9a8b”}
java.lang.IllegalArgumentException: Tiling cannot happen accross the channel dimension. The tile number of channels (3) must be the same as the image number of channels (1).
at io.bioimage.modelrunner.bioimageio.tiling.TileMaker.validateTileVsImageChannel(TileMaker.java:209)
at io.bioimage.modelrunner.bioimageio.tiling.TileMaker.validate(TileMaker.java:71)
at io.bioimage.modelrunner.bioimageio.tiling.TileMaker.(TileMaker.java:48)
at io.bioimage.modelrunner.bioimageio.tiling.TileMaker.build(TileMaker.java:62)
at io.bioimage.modelrunner.model.Model.runBMZ(Model.java:613)
at io.bioimage.modelrunner.model.Model.runBMZ(Model.java:575)
at deepimagej.Runner.run(Runner.java:138)
at deepimagej.gui.Gui.lambda$runModel$17(Gui.java:273)
at java.lang.Thread.run(Thread.java:750)

I am using the following model: https://github.com/instanseg/instanseg/releases/tag/instanseg_models_v1

I tried running on an image before and after splitting channels and still get the same error.

Any insight into what needs to be amended?

@carlosuc3m
Copy link
Collaborator

@pr4deepr
How are you getting the errors?
I just downlaoded both models, tried on the sample images and it is workin.

@abadgerw
Copy link

@carlosuc3m Hmm. I downloaded the brightfield model and placed it in the models folder. It shows up and when I click run on the following image (https://drive.google.com/file/d/1USnhzYxHRQTWyxrBUsXPoE4IHTnejQm3/view?usp=sharing) I get the error above. I tried it on the native image and following splitting of channels. It sounds like it could be something with the image input if it works for you on the sample images?

@pr4deepr
Copy link
Author

Hi
I was going through the messages again and I realized you may have used the wrong rdf for testing.

In the zip file from zenodo, the model for deepimagej is located at: ZeroCost_QA_bioimageio_model_export\2D_Ganglia_RGB_v2.bioimage.io.model.zip

This is the one I currently have in the model:

  - kwargs: {threshold: 0.7364705882352942}
    name: binarize
  halo:
      - 0
      - 191
      - 191
      - 0
  shape:
      offset:
        - 0
        - 0
        - 0
        - 0
      reference_tensor: input
      scale:
        - 1
        - 1
        - 1
        - 1

original rdf (above)

using newest deepimagej version (3.1.0 - SNAPSHOT) I've updated today,

on a test image: Java.lang.IllegalArgumentException: Tiling cannot happen accross the channel dimension. The tile number of channels (3) must be the same as the image number of channels (1).

rgb image: java.lang.IllegalArgumentException: Tiling cannot happen accross the channel dimension. The tile number of channels (3) must be the same as the image number of channels (1).

carlos_rdf with shape and offset
When I change the rdf.yaml to your recommendation

shape:
offset:
- 0
- 0
- 0
- 0.5 ## HERE
reference_tensor: input
scale:
- 1
- 1
- 1
- 0 ## HERE

test image: works fine, but incorrect prediction

rgb image: java.lang.IllegalArgumentException: Tiling cannot happen accross the channel dimension. The tile number of channels (3) must be the same as the image number of channels (1).

The model has been used successfully on older deepimagej version for past 7-8 months I believe, so the model is not corrupt.
Besides, I've used it for this paper and done [testing in Fig 2 F-H] (https://journals.biologists.com/jcs/article/137/20/jcs261950/362542/Gut-Analysis-Toolbox-automating-quantitative) from python in google colab. The predictions I got from this model using Python are in the same zip file you downloaded in Zenodoat : ZeroCost_QA_bioimageio_model_export\Quality Control\Prediction and the corresponding images are in this link under test.

I'm not sure if the preprocessing is being performed in deepimagej currently using the macro: per_sample_scale_range.ijm as this splits it into 3 channels and performs normalization. I don't think it makes it a composite after that, so it may be passed onto deepimagej as a 3 channel rgb image.

Cheers
Pradeep

@carlosuc3m
Copy link
Collaborator

carlosuc3m commented Nov 20, 2024

HEllo @pr4deepr and @abadgerw

The models should run ok now.

The pre- and post-processing were not being applied. Also the plugin was not converting directly from rgb color to rdg stack, that is why it seemed to have only one channel, instead of three. All that is fixed!

If you update the plugin, it should work now.

Sorry for the inconvenience!
Carlos

@carlosuc3m
Copy link
Collaborator

@pr4deepr

As you mentioned the pre-processing was not being applied.
In the new deepimagej, we do not use anymore macro pre and post-processing. They are integrated in the java code to make it faster and more reproducible.
The problem was that they were not being applied because some bug in the code. Thank you for your catch!

And specifically for your model, the result I get is not the same as the sample image, because the sample image was obtained without halo (without tiling).

It seems that your model is quite reactive to tiling, so depending on the halo and image size,one result or another will be obtained.

Please let me know if i can help you with anything else.

Also regarding the macro, the plugin is not recordable anymore, but there is a new plugin that automatically generates the macro if you want.
The new deepImageJ can be applied to every image in a folder and the results saved to the folder of interest.
This can be even done in headless mode, no need to open Fiji.

Regards,
Carlos

@pr4deepr
Copy link
Author

Hi
Thanks so much. It seems to be working now, but I cannot reproduce outputs from the previous deepimagej.
I agree that it appears to be a tiling related issue and its doing something different than the previous version.

Deepimagej 3.0.3

Image 1

image

Image 2 (download)

image

DeepImage 3.1.0

Image 1

image

Image 2

image

Cheers
Pradeep

@carlosuc3m
Copy link
Collaborator

HEllo @pr4deepr
It is definitely a tiling issue. Were you using the same rdf.yaml in previous versions of deepimagej?
And the tiling parameters?
Right now the tiling is defined by the software. This way, the user is guaranteed to get the same results always, and it is also way easier to understand.
I am considering an advanced mode with the option of modifying it, but it will take some time to arrive.
Regards,
Carlos

@abadgerw
Copy link

Thanks, @carlosuc3m! I tried and when it runs all I get is a black screen as an output:

Screenshot 2024-11-21 at 10 37 23 AM

When I try on the sample image, I also don't get a masked segmentation as an output:

Screenshot 2024-11-21 at 10 39 03 AM

Any insight into where I may be going wrong?

@carlosuc3m
Copy link
Collaborator

have you tried updating the plugin?
@abadgerw

@abadgerw
Copy link

@carlosuc3m yes that is why I don't get the tiling error anymore but now I'm running into the blank screen with no segmentation.

@carlosuc3m
Copy link
Collaborator

could you provide the image please? @abadgerw

@abadgerw
Copy link

abadgerw commented Nov 21, 2024

@carlosuc3m here is my image: https://drive.google.com/file/d/1USnhzYxHRQTWyxrBUsXPoE4IHTnejQm3/view?usp=sharing

I also see the same thing with the test image that comes with the instanseg model: https://github.com/instanseg/instanseg/releases/tag/instanseg_models_v1

@carlosuc3m
Copy link
Collaborator

This is the result i get with your sample image @abadgerw
image

@carlosuc3m
Copy link
Collaborator

carlosuc3m commented Nov 21, 2024

Just downloading your image and runnning with the latest updated version of deepIamgeJ.
Maybe try image>adjust>brightness/contrast to see if your histogram is not adjusted, but that should not happen.

@abadgerw
Copy link

abadgerw commented Nov 21, 2024

Thanks, @carlosuc3m. I uninstalled and reinstalled deepimageJ and got the same black screen. I also tried the adjustment of brightness/contrast which didn't seem to change the outcome.

One thought: How should I be loading this custom model? I originally placed the unzipped file in the following folder: /Applications/Fiji.app/models. It shows up when running deepimageJ. However, in case that is the issue, instead, I tried using the install model manually command and that doesn't seem to work.

I am using an Apple M1 Pro in case that is useful information.

This is what I see in the console:

Model loading: 1
Engines loading: 1
Set size: 0
Title panel: 112
Search bar: 42
Content panel: 93
Footer: 22
Model loading: 0
Engines loading: 1
Set size: 0
Title panel: 55
Search bar: 31
Content panel: 234
Footer: 3
[SERVICE-0] {"task":"9eedb080-67d6-40ac-9441-fc72df0991bd","requestType":"EXECUTE","inputs":{"modelSource":"/Applications/Fiji.app/models/brightfield_nuclei/instanseg.pt","modelFolder":"/Applications/Fiji.app/models/brightfield_nuclei"},"script":"loadModel"}
[SERVICE-0] {"responseType":"LAUNCH","task":"9eedb080-67d6-40ac-9441-fc72df0991bd"}
[SERVICE-0] {"responseType":"COMPLETION","task":"9eedb080-67d6-40ac-9441-fc72df0991bd"}
[SERVICE-0] {"task":"ca8d3151-8ba5-4658-8a02-188b6be669ac","requestType":"EXECUTE","inputs":{"outputs":["{"memoryName":"/shm-47fecf52-318d-4073-86de-a","isInput":false,"shape":[1,1,1056,1408],"name":"instance","dtype":"float32"}"],"inputs":["{"memoryName":"/shm-6ffb4dfe-4a83-4fd8-a809-c","shape":[1,3,1056,1408],"isInput":true,"name":"raw","dtype":"float32"}"]},"script":"inference"}
[SERVICE-0] {"responseType":"LAUNCH","task":"ca8d3151-8ba5-4658-8a02-188b6be669ac"}
[WORKER-0] [W SparseCsrTensorImpl.cpp:56] Warning: Sparse CSR tensor support is in beta state. If you miss a functionality in the sparse tensor support, please submit a feature request to https://github.com/pytorch/pytorch/issues. (function operator())
[SERVICE-0] {"responseType":"COMPLETION","task":"ca8d3151-8ba5-4658-8a02-188b6be669ac"}
[SERVICE-0] {"task":"71ed9c17-d55b-478b-96af-3c0a5844d17a","requestType":"EXECUTE","inputs":{"outputs":["{"memoryName":"/shm-55d21846-ec11-436f-9805-4","isInput":false,"shape":[1,1,1056,1408],"name":"instance","dtype":"float32"}"],"inputs":["{"memoryName":"/shm-9883e7c8-aca5-4930-b71e-e","shape":[1,3,1056,1408],"isInput":true,"name":"raw","dtype":"float32"}"]},"script":"inference"}
[SERVICE-0] {"responseType":"LAUNCH","task":"71ed9c17-d55b-478b-96af-3c0a5844d17a"}
[SERVICE-0] {"responseType":"COMPLETION","task":"71ed9c17-d55b-478b-96af-3c0a5844d17a"}
[SERVICE-0] {"task":"253b4fa5-3bb6-4620-a22e-9f759c8bf0d2","requestType":"EXECUTE","inputs":{"outputs":["{"memoryName":"/shm-6d2763cd-3d92-4310-a054-1","isInput":false,"shape":[1,1,1056,1408],"name":"instance","dtype":"float32"}"],"inputs":["{"memoryName":"/shm-301c1ff2-a635-4a48-bc97-5","shape":[1,3,1056,1408],"isInput":true,"name":"raw","dtype":"float32"}"]},"script":"inference"}
[SERVICE-0] {"responseType":"LAUNCH","task":"253b4fa5-3bb6-4620-a22e-9f759c8bf0d2"}
[SERVICE-0] {"responseType":"COMPLETION","task":"253b4fa5-3bb6-4620-a22e-9f759c8bf0d2"}
[SERVICE-0] {"task":"25c802fb-ec1f-4b0c-a7b2-857f72d8a4e2","requestType":"EXECUTE","inputs":{"outputs":["{"memoryName":"/shm-b5ed02e5-481f-4ae6-9d95-b","isInput":false,"shape":[1,1,1056,1408],"name":"instance","dtype":"float32"}"],"inputs":["{"memoryName":"/shm-0e40bde4-73de-4b1c-9e22-e","shape":[1,3,1056,1408],"isInput":true,"name":"raw","dtype":"float32"}"]},"script":"inference"}
[SERVICE-0] {"responseType":"LAUNCH","task":"25c802fb-ec1f-4b0c-a7b2-857f72d8a4e2"}
[SERVICE-0] {"responseType":"COMPLETION","task":"25c802fb-ec1f-4b0c-a7b2-857f72d8a4e2"}

And this is what my screen looks like:

Screenshot 2024-11-21 at 11 53 11 AM

@carlosuc3m
Copy link
Collaborator

HEllo @abadgerw
Wow this is quite surprising, but it seems that something might not be correct with MAcs.
I only ahve access to Windows and linux machines.
I will try to reproduce this and get back to you.
Thanks for noticing!

Maybe @cfusterbarcelo can try to reproduce this?

@pr4deepr
Copy link
Author

HEllo @pr4deepr It is definitely a tiling issue. Were you using the same rdf.yaml in previous versions of deepimagej? And the tiling parameters? Right now the tiling is defined by the software. This way, the user is guaranteed to get the same results always, and it is also way easier to understand. I am considering an advanced mode with the option of modifying it, but it will take some time to arrive. Regards, Carlos

This is an older model I used in older deepimagej version:

https://github.com/pr4deepr/GutAnalysisToolbox/blob/v0.4/Models/2D_enteric_ganglia_v2.bioimage.io.model.zip

link to model_yaml. in this file

I think the settings were similar.

Cheers
Pradeep

@abadgerw
Copy link

Thanks, @carlosuc3m and @cfusterbarcelo! Do not hesitate to let me know if you require anything on my end as you troubleshoot.

@carlosuc3m
Copy link
Collaborator

It should work now @abadgerw .
I think I have fixed that error an many more that appeared along the way 🤣

@abadgerw
Copy link

Thanks, @carlosuc3m! I tried again and look to be still having the same issue. The console outputs the following:

[SERVICE-0] {"task":"cb059d92-fff1-47c5-9539-bfbcec019f2d","requestType":"EXECUTE","inputs":{"modelSource":"/Applications/Fiji.app/models/brightfield_nuclei/instanseg.pt","modelFolder":"/Applications/Fiji.app/models/brightfield_nuclei"},"script":"loadModel"}
[SERVICE-0] {"responseType":"LAUNCH","task":"cb059d92-fff1-47c5-9539-bfbcec019f2d"}
[WORKER-0] Warning: Versions of org.bytedeco:javacpp:1.5.9 and org.bytedeco:openblas:0.3.21-1.5.8 do not match.
[SERVICE-0] {"responseType":"COMPLETION","task":"cb059d92-fff1-47c5-9539-bfbcec019f2d"}
[SERVICE-0] {"task":"6fdfe6b7-45bb-41fa-834b-6f20ad8d78ab","requestType":"EXECUTE","inputs":{"outputs":["{"memoryName":"/shm-ca56e0cc-a3b7-40e8-b198-c","isInput":false,"shape":[1,1,1056,1408],"name":"instance","dtype":"float32"}"],"inputs":["{"memoryName":"/shm-2fac58f1-8f13-4e8d-8800-c","shape":[1,3,1056,1408],"isInput":true,"name":"raw","dtype":"float32"}"]},"script":"inference"}
[SERVICE-0] {"responseType":"LAUNCH","task":"6fdfe6b7-45bb-41fa-834b-6f20ad8d78ab"}
[WORKER-0] [W SparseCsrTensorImpl.cpp:56] Warning: Sparse CSR tensor support is in beta state. If you miss a functionality in the sparse tensor support, please submit a feature request to https://github.com/pytorch/pytorch/issues. (function operator())
[SERVICE-0] {"responseType":"COMPLETION","task":"6fdfe6b7-45bb-41fa-834b-6f20ad8d78ab"}
[SERVICE-0] {"task":"566948ef-3848-4359-9a54-799c1aa92170","requestType":"EXECUTE","inputs":{"outputs":["{"memoryName":"/shm-3cff63d9-951e-45b2-ae6e-8","isInput":false,"shape":[1,1,1056,1408],"name":"instance","dtype":"float32"}"],"inputs":["{"memoryName":"/shm-fa2c1ba0-dd67-42e4-bbd3-0","shape":[1,3,1056,1408],"isInput":true,"name":"raw","dtype":"float32"}"]},"script":"inference"}
[SERVICE-0] {"responseType":"LAUNCH","task":"566948ef-3848-4359-9a54-799c1aa92170"}
[SERVICE-0] {"responseType":"COMPLETION","task":"566948ef-3848-4359-9a54-799c1aa92170"}
[SERVICE-0] {"task":"d20756da-ba7b-41d4-8d8c-6e77cb30cda9","requestType":"EXECUTE","inputs":{"outputs":["{"memoryName":"/shm-b2642822-9d58-4751-8607-3","isInput":false,"shape":[1,1,1056,1408],"name":"instance","dtype":"float32"}"],"inputs":["{"memoryName":"/shm-6dd2b928-8c1a-4d54-bc96-9","shape":[1,3,1056,1408],"isInput":true,"name":"raw","dtype":"float32"}"]},"script":"inference"}
[SERVICE-0] {"responseType":"LAUNCH","task":"d20756da-ba7b-41d4-8d8c-6e77cb30cda9"}
[SERVICE-0] {"responseType":"COMPLETION","task":"d20756da-ba7b-41d4-8d8c-6e77cb30cda9"}
[SERVICE-0] {"task":"b1b4b96d-2e92-4507-8d41-094189e4f509","requestType":"EXECUTE","inputs":{"outputs":["{"memoryName":"/shm-6eab8bb6-59f6-4043-80c6-4","isInput":false,"shape":[1,1,1056,1408],"name":"instance","dtype":"float32"}"],"inputs":["{"memoryName":"/shm-c0e1c123-7d49-4c2b-8ea0-5","shape":[1,3,1056,1408],"isInput":true,"name":"raw","dtype":"float32"}"]},"script":"inference"}
[SERVICE-0] {"responseType":"LAUNCH","task":"b1b4b96d-2e92-4507-8d41-094189e4f509"}
[SERVICE-0] {"responseType":"COMPLETION","task":"b1b4b96d-2e92-4507-8d41-094189e4f509"}

And the output file is the following:

Screenshot 2024-11-25 at 4 18 34 PM

@carlosuc3m
Copy link
Collaborator

@abadgerw
could you please show me the list of files inside your
/Applications/Fiji.app/engines/pytorch folders?
There should be two folders called something like: pytorch-1.13.1-macosx-x64_86 and pytorch-1.13.1-macosx-x64_86

@abadgerw
Copy link

@carlosuc3m

Screenshot 2024-11-25 at 5 17 26 PM Screenshot 2024-11-25 at 5 17 32 PM

@carlosuc3m
Copy link
Collaborator

what about pytorch 2?

@abadgerw
Copy link

Screenshot 2024-11-25 at 5 33 12 PM

@carlosuc3m
Copy link
Collaborator

Just some heads up that I have not forgotten you and I am on i!
@pr4deepr
@abadgerw

@alanocallaghan

This comment was marked as resolved.

@carlosuc3m
Copy link
Collaborator

HEllo again @abadgerw
I have added some prints to the code, could you please update deepimagej through the update site, run it again and paste the input here, just to see what is happening?

sorry to bother again!

Also @pr4deepr I am working on the issue but i am quite busy too!

REgards,
Carlos

@abadgerw
Copy link

abadgerw commented Dec 9, 2024

@carlosuc3m apologies for the delay. Here are the console outputs when running the updated plugin (the screen still show a black screen after running the model):

[SERVICE-0] {"task":"12f3f646-08e8-473f-8e3b-0ff620a21f0f","requestType":"EXECUTE","inputs":{"modelSource":"/Applications/Fiji.app/models/brightfield_nuclei/instanseg.pt","modelFolder":"/Applications/Fiji.app/models/brightfield_nuclei"},"script":"loadModel"}
[SERVICE-0] {"responseType":"LAUNCH","task":"12f3f646-08e8-473f-8e3b-0ff620a21f0f"}
[WORKER-0] Warning: Versions of org.bytedeco:javacpp:1.5.8 and org.bytedeco:pytorch:2.0.1-1.5.9 do not match.
[SERVICE-0] {"responseType":"COMPLETION","task":"12f3f646-08e8-473f-8e3b-0ff620a21f0f"}
[SERVICE-0] {"task":"14b159ec-01ee-4b86-abfa-ea938fd7203a","requestType":"EXECUTE","inputs":{"outputs":["{"memoryName":"/shm-e9b44327-82ab-4410-81b9-7","isInput":false,"shape":[1,1,1056,1408],"name":"instance","dtype":"float32"}"],"inputs":["{"memoryName":"/shm-d59f3af6-d187-4c6b-9937-0","shape":[1,3,1056,1408],"isInput":true,"name":"raw","dtype":"float32"}"]},"script":"inference"}
[SERVICE-0] {"responseType":"LAUNCH","task":"14b159ec-01ee-4b86-abfa-ea938fd7203a"}
[SERVICE-0] Input max: 3.1003989471563117E38
[WORKER-0] [W SparseCsrTensorImpl.cpp:56] Warning: Sparse CSR tensor support is in beta state. If you miss a functionality in the sparse tensor support, please submit a feature request to https://github.com/pytorch/pytorch/issues. (function operator())
[SERVICE-0] {"responseType":"COMPLETION","task":"14b159ec-01ee-4b86-abfa-ea938fd7203a"}
Output SHM 0 max value: 1.0
Copied ouput 0 max value: 1.0
[SERVICE-0] {"task":"f30e373a-781f-4a03-99a0-2ce353ac5302","requestType":"EXECUTE","inputs":{"outputs":["{"memoryName":"/shm-dcb4e9d9-ebf4-45a6-b308-0","isInput":false,"shape":[1,1,1056,1408],"name":"instance","dtype":"float32"}"],"inputs":["{"memoryName":"/shm-e4ffa7e4-67a6-4ce4-9b0e-1","shape":[1,3,1056,1408],"isInput":true,"name":"raw","dtype":"float32"}"]},"script":"inference"}
[SERVICE-0] {"responseType":"LAUNCH","task":"f30e373a-781f-4a03-99a0-2ce353ac5302"}
[SERVICE-0] Input max: 3.1003989471563117E38
[SERVICE-0] {"responseType":"COMPLETION","task":"f30e373a-781f-4a03-99a0-2ce353ac5302"}
Output SHM 0 max value: 2.0
Copied ouput 0 max value: 2.0
[SERVICE-0] {"task":"7e92b210-b518-4fb3-9ba6-9831662b5103","requestType":"EXECUTE","inputs":{"outputs":["{"memoryName":"/shm-8871442d-e062-46f2-b00c-2","isInput":false,"shape":[1,1,1056,1408],"name":"instance","dtype":"float32"}"],"inputs":["{"memoryName":"/shm-809f8dc1-ac9d-445e-a2d7-3","shape":[1,3,1056,1408],"isInput":true,"name":"raw","dtype":"float32"}"]},"script":"inference"}
[SERVICE-0] {"responseType":"LAUNCH","task":"7e92b210-b518-4fb3-9ba6-9831662b5103"}
[SERVICE-0] Input max: 3.1003989471563117E38
[SERVICE-0] {"responseType":"COMPLETION","task":"7e92b210-b518-4fb3-9ba6-9831662b5103"}
Output SHM 0 max value: 0.0
Copied ouput 0 max value: 0.0
[SERVICE-0] {"task":"a6983b02-908e-48a6-83bd-ec11ae35d7d1","requestType":"EXECUTE","inputs":{"outputs":["{"memoryName":"/shm-1d74236c-e98e-4cf2-bb58-4","isInput":false,"shape":[1,1,1056,1408],"name":"instance","dtype":"float32"}"],"inputs":["{"memoryName":"/shm-a89a921d-1e96-4241-9fc5-2","shape":[1,3,1056,1408],"isInput":true,"name":"raw","dtype":"float32"}"]},"script":"inference"}
[SERVICE-0] {"responseType":"LAUNCH","task":"a6983b02-908e-48a6-83bd-ec11ae35d7d1"}
[SERVICE-0] Input max: 3.1003989471563117E38
[SERVICE-0] {"responseType":"COMPLETION","task":"a6983b02-908e-48a6-83bd-ec11ae35d7d1"}
Output SHM 0 max value: 1.0
Copied ouput 0 max value: 1.0

@carlosuc3m
Copy link
Collaborator

HEllo @abadgerw I am back!!
Sorry for the late reply, I have been busy and the mac error was quite tricky.
Howeer, it should be solved now! Thanks a lot for all the help and feedback.

I need to update the deepImageJ update site and once it is done, I will let you know and if you want you can try it!

Thanks a lot!
Carlos

@carlosuc3m
Copy link
Collaborator

Also @pr4deepr sorry for the delay with your issue. Now that this is fixed, I am fully back on the tiling thing!
Sorry again!
Carlos

@pr4deepr
Copy link
Author

Thanks for letting me know!

@carlosuc3m
Copy link
Collaborator

@abadgerw it should work now!!

@carlosuc3m
Copy link
Collaborator

HEllo @pr4deepr issue identified!
Trying to fix it now.

@abadgerw
Copy link

Thanks, @carlosuc3m! I now see an output!

Screenshot 2024-12-20 at 12 40 29 PM

However, I was wondering why the image has a gradient (dark at the top with nothing detected)? It seems to be a different output compared to when I use this same model directly inside QuPath.

@carlosuc3m
Copy link
Collaborator

HEllo @pr4deepr.
Even though I have not updated as much, I have been trying to understand the issue.
And finally i can say that I have found the problem (I hope there are no more issues down the line😬).
The problem was related to tiling and how imglib2 handles mirroring.

If no more issues arise, a new dij version will be released today or tomorrow.
As a byproduct, your issue should be resolved too @abadgerw . It would be interesting if you could provide me with the qupath output image.

Thanks for your patience and happy new year!!

Regards,
Carlos

@carlosuc3m
Copy link
Collaborator

Hello again @pr4deepr @abadgerw , forgot to mention but you now should be able to update deepimagej.

Regards,
Carlos

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

4 participants