Support for QuPath 0.5 and local models

[alanocallaghan]

Resolves #32

[alanocallaghan]

Local model support implemented, whenever the user directory is changed we scan for directories, and add them based on the folder name. Invalid entries pop up a notification, otherwise models should work as for online models, at least based on my running a modified version of a remote model

Disabled checking modification times and checksums because that seems unnecessary for user models.

[petebankhead]

@alanocallaghan can you briefly explain how to use this?

Put what files where...?

You mention 'whenever the user directory is changed' but I assume that's not referring to QuPath's 'user directory' - right?

[alanocallaghan]

Feels like a picture paints a thousand words. my_model pops up in the searchable combobox and works like any other model

[alanocallaghan]

The "user directory" refers to the "User model directory" in the WSInfer extension window

[kaczmarj]

fantastic! 🚀

[xalexalex]

Fantastic indeed! Can confirm this also works with home-built models and custom config.json. Another great upgrade—thanks!

[petebankhead]

Great that it works!

Question to all: Would it be a good or bad idea to just use one model directory, and not store user models somewhere else?

The main reasons for:

1. Simpler
2. On a small (e.g. 13" laptop) monitor, the current dialog is sometimes too tall... so we need to solve a vertical space problem if we can't shrink it somehow

Main reason against (that I can think of):

1. Potentially overwriting user models if there are name clashes

[alanocallaghan]

Great that it works!

Question to all: Would it be a good or bad idea to just use one model directory, and not store user models somewhere else?

The main reasons for:

1. Simpler
2. On a small (e.g. 13" laptop) monitor, the current dialog is sometimes too tall... so we need to solve a vertical space problem if we can't shrink it somehow

Main reason against (that I can think of):

1. Potentially overwriting user models if there are name clashes

Could just have a "user" subdirectory thats checked automatically, and ensure no model named user is ever added to the model zoo

[xalexalex]

Personally a single directory is what I was expecting. In fact I had added my models to an acaputo directory that was a sibling of the kaczmarj directory for the default (zoo) models.

Regarding your concern:

2. On a small (e.g. 13" laptop) monitor, the current dialog is sometimes too tall

If I understood correctly, I don't think this is going to be much of a problem because I would advise anyone with more than 6–8 models to use the text filtering capabilities of that dropdown.

[kaczmarj]

Question to all: Would it be a good or bad idea to just use one model directory, and not store user models somewhere else?

i think it is a good idea to have just one model directory, because then there will be a single source of truth. this would simplify the user experience imho.

if this isn't part of the extension yet, it might be useful to have a button that opens a file browser to the model directory. although i hope users wouldn't fiddle around with the downloaded models, which could break something ;)

i wanted to add some info to one point:

In fact I had added my models to an acaputo directory that was a sibling of the kaczmarj directory for the default (zoo) models.

just for clarity, the kaczmarj directory is created because that's my huggingface username, and all of the models in the zoo so far are on my huggingface account.

[vipulnj]

Thank you all for your efforts in this direction. Not sure if this feature is done, but I am assuming it is, since it has been merged and a release has been made.

I just tried using wsinfer (v0.3) on the the latest qupath (v0.5) with my local models (.pt models, trained on CUDA) on my computer (mac).

• But I do not see my local models ( looking for the dir names which they are in) in the dropdown.
• I do not see a way to set "user model directory" too. The box is missing.
• I also tried re-specifying the "downloaded model directory" but don't see my local models in the dropdown. Even restarting qupath app did not help. Is there a way to force "refresh" the dropdown list?

In term of directory structure,

• I placed the wsinfer_local as shown in the screenshot above.
• Also, tried it moving the wsinfer_local as a sibling to existing directory (kaczmarj) under ~/QuPath/v0.5/wsinfer

Hopefully this feedback is helpful. Please let me know if I'm doing something wrong and there is a way to fix this.

[alanocallaghan]

Hey @vipulnj, indeed the steps for this process have changed since this pull request. The updated process is detailed in #48, basically:

• Find the downloaded model directory
• Create a "local" or "user" directory in that folder
• Add models as subdirectories of the "local"/"user" directory
• Hey presto

Do let us know if there's any issues with that process

[sandyCarmichael]

This is a fantastic feature!

Are there any plans to include this on the Qupath site or the Use your own model section of the WSInfer site? Or is the documented elsewhere and I have just missed it?

[kaczmarj]

it isn't on the qupath site, but it should be. @sandyCarmichael - would you be interested in submitting a pr that adds these steps to the qupath documentation?

[MITEL-UNIUD]

Hey @vipulnj, indeed the steps for this process have changed since this pull request. The updated process is detailed in #48, basically:

• Find the downloaded model directory
• Create a "local" or "user" directory in that folder
• Add models as subdirectories of the "local"/"user" directory
• Hey presto

Do let us know if there's any issues with that process

Hi, I am on Mac M1, Qupath 0.5.1, WSInfer 0.3. I tried to add a local model (which I tested on WSInfer command line) but no success. I tried local and user both as sibling and as descendant of kaczmarj in the models directory. I also restarted Qupath after every change. What I am doing wrong? thanks.

[alanocallaghan]

Hi, I am on Mac M1, Qupath 0.5.1, WSInfer 0.3. I tried to add a local model (which I tested on WSInfer command line) but no success. I tried local and user both as sibling and as descendant of kaczmarj in the models directory. I also restarted Qupath after every change. What I am doing wrong? thanks.

The model should definitely be as a sibling of kaczmarj. Can you provide a screenshot of the model directory, showing the overall structure of the user model directories and files?

[MITEL-UNIUD]

Here it is, thanks. I also tried to have the main folder inside the model folder.

[alanocallaghan]

Do you see any messages in the log viewer when opening WSInfer with this setup?

[MITEL-UNIUD]

Apologies for not having checked the log before (never had need, I did even not know where to look for :) ). Yes, it expects a model specifically named torchscript_model.pt . After having renamed it, it is recognised.
However, then I have another issue so the model is not running. This is notified by a WSinfer dialog as follows (it is much longer, I even cannot read it all). Maybe it has to do with how I created it, however it runs from command line. There are two warnings in the log also:

11:58:06.66	[JavaFX Application Thread]	WARN	qupath.ext.wsinfer.models.WSInferUtils	Replaced model breastcancer with local version
11:58:16.405	[wsinfer1]	INFO	qupath.ext.wsinfer.WSInfer	Attempting to set device to mps
11:58:16.406	[wsinfer1]	WARN	ai.djl.repository.SimpleRepository	Simple repository pointing to a non-archive file.
11:58:16.432	[Finalizer]	WARN	ai.djl.BaseModel	Model: torchscript_model was not closed explicitly.

[alanocallaghan]

This error is vaguely familiar but I can't remember exactly what caused it before. There are some DJL and Pytorch related issues:

• DJL load model in cpu throw error: Could not run aten::empty_strided with arguments from the CUDA backend deepjavalibrary/djl#2474
• Could not run 'aten::empty_strided' with arguments from the 'CUDA' backend pytorch/pytorch#71402

that might help?

[MITEL-UNIUD]

Thanks, not yet able to solve that, anyway I submitted a pull request for the docs related to the Qupath extension so the basics are documented.

[alanocallaghan]

Thanks! Alternative may be to run the model on CPU for now

[MITEL-UNIUD]

well, not really: also on CPU it does not run, same error. (Zoo models run).
I am not in a hurry, I will find a solution.

[kaczmarj]

can you try transferring your model to cpu before exporting it to torchscript? some pseudo-code to illustrate what i am thinking:

model.eval()
model.cpu()

# Add other arguments as needed.
model_jit = torch.jit.script(model)

torch.jit.save(model_jit, "torchscript_model.pt")

then i think the model should work on cpu or gpu.... fingers crossed 🤞

[MITEL-UNIUD]

@kaczmarj : thanks, this plus some magic found on Stack Overflow solved the issue (I have yet to consolidate & understand what I did, but now it runs :) ).