Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Added Live Preview via overlays #20

Open
wants to merge 13 commits into
base: main
Choose a base branch
from

Conversation

rmdocherty
Copy link

Hi @Karol-G,

I was working on some Napari-SAM related stuff and got a 'live-view' working (where it prompts SAM at the cursor to show the label about to be placed).
napari_sam_live_preview

Main changes:

  1. Live preview - this uses a custom Napari overlay with associated custom vispy overlay that displays an ImageNode that is attached to the Image Layer. When the cursor is moved and the time between the last prediction is large enough (to avoid over-pinging) SAM is prompted with the current position of the mouse cursor and previous points. The returned mask is then converted to an image, cropped to a bounding box around the positive part of the mask (to avoid redrawing an image the size of the canvas to screen all the time) and the ImageNode's data updated. This live preview also works in bounding box mode.
  2. Bounding box selector is now an overlay using one of the default overlays, the 'selection_box' overlay. The width and colour are set as before. However, this means the bounding boxes no longer persist after a new one is added. This means one of the 'ignore_this_layer' has been removed - there is the potential to remove the second but that would probably require another custom points overlay.

Issues:

  1. The biggest is that this PR uses private attribute access to add the custom overlays to the image layer - this gives a deprecation warning for version 5.0.0. I'm assuming (hoping) that there will still be a way to add/change overlays after that update. The overlays API is probably unstable and may change - I've stuck with fairly standard stuff that is used elsewhere though.
  2. It doesn't work with 3D images - unsure how x_coord parameter really works so have left it out.
  3. The initial transformation copying in _custom_overlays is very hacky - I'm not too sure on the whole rendering pipeline but this seems to work.
  4. I had a weird bug with the 'edge_color' argument of add_points so removed it - this might just be my napari version.

@Karol-G
Copy link
Collaborator

Karol-G commented Mar 18, 2024

Hey @rmdocherty,

This looks really cool! However, I have to give you sadly the same response as on another pull request -> I am currently in a crunch time phase and won't be able to review this PR for a couple of weeks, as other projects have a higher priority right now. I will definitely take a look at it though and will set a reminder so I don't forget. I hope this is okay with you :)

On a side note, the image looks like a CT mineral particle sample? Is that correct? Coincidently, I created an automatic segmentation method for these type of images called ParticleSeg3D. Maybe this is interesting to you :)

GitHub: (https://github.com/MIC-DKFZ/ParticleSeg3D)
Paper (Arxiv version): https://arxiv.org/abs/2301.13319

Best regards,
Karol

@Karol-G Karol-G self-assigned this Mar 20, 2024
@rmdocherty
Copy link
Author

No worries! The image is a CT of catalyst particles (though I mainly do batteries), so similar to mineral samples - the framework looks interesting and I will try it out at some point

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants