Prompting with vague segmentation mask

[mano3-1]

Hi,

I tried prompting the model with a vague segmentation mask that I obtained from another model. The prompt isn’t very accurate, but it at least covers the object of interest. I expected SAM2 to refine the mask more precisely, but the results are not better.

Here is the mask prompt:

And result from SAM:

There are small patches scattered on the right side, and the left side of the mask is completely off.

What could be the reason? Does SAM2 always need a point/box to refine the masks?

[Caspeerrr]

One thing you could try is transforming the segmentation mask to a bounding box and using that as input to sam2:
torchvision.ops.masks_to_boxes

[chayryali]

This behavior is not surprising because the mask prompt here spans multiple objects. Prompts are expected to correspond to one object (or a part or sub-part of an object) but not multiple objects.

Can you try prompting with mask prompts corresponding to one object?